Is this a worthwhile idea for a program?

[mouser]

Well if does what we're talking about then it does what we're talking about

[IainB]

This might be of help/use: For years I have used a brilliant text parser/"cleaner" for jokes that I send out to subscribers via my jokes listserver (which draws on a large database of collected jokes that people have sent me via email).

I often have to reformat the jokes first, because they need it. I use a freeware program called Cleaner (version 1.02), which was introduced to me by an American guy I used to work with on a project in Thailand (Hi Ed!), and which is a real time-saver.  This utility is ideal if you wish to keep and use text received through email.  It was really annoying to see all the ">" symbols in front of each line - though these have become much less common with today's technology.  Not only does this nifty little program remove them from email messages, making the message look cleaner and preventing choppy word wraps, but also it clears any special formatting properties - e.g., losing any special formats that may have been put in by Outlook or Word. This is quite handy nowadays when people seem to use all sorts of large text formats and insert irrelevant GIFs into the joke.
For example:

The cleaner helps to just strip out the bare text and make it more legible.
However, it does not help to make the jokes any funnier.    

If you want a copy of Cleaner to have a look-see, you can get it (the latest version is called eCleaner version 2.02) from various sites on the Internet.  The author's website is now http://ecleaner.tripod.com/.
This version has many more features - for example, it has buttons to strip out HTML code from the text.

eCleaner probably wouldn't be of much use for the purposes of this discussion thread (text language syntax parsing), but I suppose you could do worse than ask the author of eCleaner if you could have the code to adapt to parsing different languages. (Why re-invent all parts of the wheel?) He might even be interested in helping out - his website indicates that eCleaner is becoming obsolete, so he might like to collaborate with DC coders in a new challenge. His email contact details are in the Help file for eCleaner.

Update: I have sent the author of eCleaner an email with the contents of this post, so he can see for himself whether he likes the idea.

[MilesAhead]

Well if does what we're talking about then it does what we're talking about

-mouser (October 08, 2011, 05:49 PM)

Windows users generally aren't comfortable using a chain of filters.  Everything has to happen in one module.  Some people won't even use a program if it uses more than one process or even a dll(that they notice. When pointed out that most every exe uses one or more Windows system dlls usually a disappointed silence follows.)  Unix based systems have solved most of these issues a long time ago.  But that's why clipboard is there. Have at it.

If it was me I'd investigate Windows versions of Linux/Unix stream editors and look for already debugged scripts.

[rjbull]

text, hit the hotkey. Hit Go to run the script in the editor.  What's the difference?

-MilesAhead (October 08, 2011, 04:06 PM)

Yes, you get the same effect, but you get it in less keystrokes/mouse clicks.  That can really add up.

[superboyac]

Well if does what we're talking about then it does what we're talking about

-mouser (October 08, 2011, 05:49 PM)

Windows users generally aren't comfortable using a chain of filters.  Everything has to happen in one module.  Some people won't even use a program if it uses more than one process or even a dll(that they notice. When pointed out that most every exe uses one or more Windows system dlls usually a disappointed silence follows.)  Unix based systems have solved most of these issues a long time ago.  But that's why clipboard is there. Have at it.

If it was me I'd investigate Windows versions of Linux/Unix stream editors and look for already debugged scripts.

-MilesAhead (October 09, 2011, 01:41 PM)

Dang, this psychology is right on!

[MilesAhead]

text, hit the hotkey. Hit Go to run the script in the editor.  What's the difference?

-MilesAhead (October 08, 2011, 04:06 PM)

Yes, you get the same effect, but you get it in less keystrokes/mouse clicks.  That can really add up.

-rjbull (October 09, 2011, 02:30 PM)

You may be right. When it comes to substitutions such as regex or perl type business or even sed and awk I try to find stuff already tested.  I find figuring that stuff out more difficult than the rest of the program often times.  Everyone has their aptitudes.  One of mine is definitely eating cheese pastry.  But I don't think that will help in this case.

I started with Dos so I was never fully comfortable with Linux. Mainly due to lack of "Windowsie" editors.  But I have to give credit when it comes to using prefabricated stuff slapped together to get stuff done, esp. with text.

[IainB]

@rjbull:

Maybe IainB hasn't discovered it yet   

Heh. That's true - I don't understand (have not "discovered") a great deal of functionality in CHS nor the requirements that could have conceivably led to them. To be fair, I did at least make provision for them in some general and vague lines in the requirements analysis spreadsheet though.
There's still a lot in CHS that I have not yet managed to wrap my head around.
As @superboyac puts it:

mouser, I didn't even realize CHS could do this until maybe last week or so.  Guys, check it out, it's freaking awesome:

I already think that what @mouser has done so far with CHS as a relational database is pretty impressive.

[mouser]

I've uploaded an alpha version of the new CHS which now lets you configure external tools associated with a formatting preset, to do what we talked about here; more info and download: https://www.donation....msg264724#msg264724

note that this is NOT unicode compatible, so doesn't solve the original request.

[rjbull]

When it comes to substitutions such as regex or perl type business or even sed and awk I try to find stuff already tested.  I find figuring that stuff out more difficult than the rest of the program often times.
[...]
Linux. [...] I have to give credit when it comes to using prefabricated stuff slapped together to get stuff done, esp. with text.

-MilesAhead (October 09, 2011, 11:24 PM)

Coloured by my own experience, I understood the OP to be suggesting a container for putting scripts of that sort in, or other text-manipulation tools, with the idea of calling them by hotkey, to directly operate on the contents of the clipboard.  You're right that figuring out the scripts and/or command lines themselves is another problem (and why I was bemused by the existence of mouser's Drag and Drop Robot).  This putative program would have absolutely the same effect as defining scripts as external tools in your favourite editor, but be a low-drag way of invoking them.

Both Clippy and Text Monkey are very capable, with a wide variety of functions.  But neither of them offer the ability to add external tools, as mouser's CHS does.

[MilesAhead]

To fully understand the task I think I'd have to see the vertical editor in use. I didn't envision running little snippets through a filter.  But running the whole file through for patterns that trigger changes.  Of course the pattern would have to be exclusive enough to avoid false matches. It may be that it's not feasible.  But my thought was more like "prepping" the text before manually working like a diner cook preps the home fries.  Put the whole file through a string of stream edits until you make all the substitutions that can be automated.  Then just edit as normal.

Trouble with the substitution thing, unless you do it constantly and are very adept, it usually involves time consuming trial and error.  That's why I try to use "canned" stream filter scripts.. esp. one-liners.

[IainB]

Still on the tack of trying to understand what there may be to learn from existing wheels, I did a Google search on "Parse and convert numeric text from English to Polish" and got an interesesting list of results. For example:

• Translation Gadgets and Software
• Free Online English to Polish Translator
• Utilities to convert dates and numbers from one human language syntax/format to another - including a Microsoft-provided VBA function for use in Excel.

Hope this is of use/help.

[tranglos]

Okay, this thread ain't dead, it just took a while before I could pop it off the stack...

cough cough..

My Clipboard Help+Spell program can do all of this already.

-mouser (October 07, 2011, 03:56 PM)

I knew that! :-)

Plus it will allow you to list your saved transformation operations in a menu that you can trigger, or assign them to specific hotkeys for really quick work.

That, I didn't know. Somehow I figured you'd have to set all the options in that dialog box every time. The ability to save presets is great, and you're really providing for pretty much everything there.

Now, I'll probably have a go at it my own way, because I'm itching to do a scripting-enabled application. Today I was almost ready to start adding it to Echo, but on second thought, I'm not certain the transformations belong in it. This is because:

(a) most of the time you don't need the transformations at all. I know I only need them when doing a specific job in some app, for an hour or a few, and even then only a small percentage of everything that's copied to clipboard needs to be transformed.

(b) transforming everything (because the feature's been turned on) wastes CPU cycles and memory (esp. extenral scripts) and could add much unneeded clips to the database (assuming for the sake of argument that every clip is stored in its original and the transformed form).

(c) Switching the transformations on and off, and selecting the one to use - that's work. The functionality needs to be instantly available when needed, and absent otherwise.

I put an example here, but you gave one already, reducing 9 operations to two. That's the ideal, but perhaps not quite attainable, since you'd have to have a separate hotkey for every script or transformation you ever apply, which is unrealistic. (But you may have hotkeys for the 10 most frequent, say.) I'll be happy with four steps:

• Select the string
• Press a hotkey to bring up the app-to-be-named or just a pop-up menu
• Select script / transformation from the menu (with instant search, most recently used scrips on top, all that goodness)
• Press Enter to execute.

That way we've just eliminated 60% of user actions, and this is a good target. It seems to me that a dedicated app can make this process smoother, more organic. It's more like a special-purpose macro utility. In fact, I suppose all capable macro apps could be used for this purpose as well, if only I could pick one

[tranglos]

You may be better off using a programmable editor like Vim.

-MilesAhead (October 07, 2011, 02:50 PM)

I wish! I could do that in EmEditor or UltraEdit as well, seeing as they both come with JavaScript built in. Trouble is, for my specific needs, I work in a highly specialized "vertical" app called Trados (most translators know (of) it; most non-translators don't). Using it is a requirement, and it is actually a good thing, but it lacks certain features. Anyway, I don't have the liberty of using an editor of choice, hence the idea for a sidekick app.

-tranglos (October 07, 2011, 03:21 PM)

I don't get it. If you can clipboard stuff out, then paste it back in after transformations, you can paste it into a programmable editor and paste it back?  Maybe I have to be there to see the obstacle.

-MilesAhead (October 07, 2011, 06:35 PM)

This is what I'm doing already. I select, copy, paste into TextPad or EmEditor, run a macro, switch back, paste...

The point is to streamline all that; particularly eliminate the app switching, and make macro selection easier and faster.

But you are right; there are already ways to do that. That's one reason I posted this as a question in the first place :-)

[tranglos]

Yes but what's the difference between a "formatting program" and an editor that has a script loaded sitting there waiting for the paste? For all purposes the editor is an interpreter running your "formatting program."

Set it as default in Text Editor Anywhere.  Select text, hit the hotkey. Hit Go to run the script in the editor.  What's the difference?

-MilesAhead (October 08, 2011, 04:06 PM)

Not much! Depends on how you look at it. I'd say one difference is focus - sometimes you can do things quicker and smoother using a dedicated tool. Such a tool is not an editor, so it can be more focused on the specific task, and in some ways do it better than an editor that has to worry about so much more.

Then there's the mechanics. If you're transforming a line of text, no difference. But very often I need to strip html or xml tags from a document so that I can spell-check it (the business app does not have a working spellchecker, either). The file may be 10 MB, most of which are tags. Sometimes there is not a single linebreak in those 10 MB of xml, because it was machine-generated and no linebreaks are needed for the document to be valid.

Now, if you'ever pasted 10 MB of XML as one line into a syntax-highlighting editor with word wrapping enabled... you know what happens. What happens depends on the editor, but typically it will chew on it for a long time before it will accept any more input from you. Especially if the syntax highlighting works from regular expressions.

Then you run the script, and it still takes more time than it has to, because the editor will maintain its undo buffer and update the display... All these can be turned off, sure. But a dedicated app doesn't need to paste the mess anywhere, doesn't need undo, doesn't refresh the screen, so it can do the work in a fraction of the time a full editor will take.

Then there are little things, like whether you can assign shortcuts to macros in the editor for faster execution. Trivial molehills that tend to become mountains when you repeat them over and over for years. Sixteen years in my case; I'm ready for an improvement  :-)

[tranglos]

This putative program would have absolutely the same effect as defining scripts as external tools in your favourite editor, but be a low-drag way of invoking them.

-rjbull (October 10, 2011, 03:40 PM)

Emphasis mine. That's exactly what I was trying to say, thanks!