Clipboard Help+Spell Extended Clipboard Format Data Discussion Thread

[mouser]

Updated with somewhat improved capture options:

[cranioscopical]

I'd be curious if folks would prefer to have image data stored in the normal clip database as well?

-mouser (February 13, 2020, 09:43 PM)

For me, keeping the database robust trumps duplication.

[djMot]

I'd be curious if folks would prefer to have image data stored in the normal clip database as well?

-mouser (February 13, 2020, 09:43 PM)

For me, keeping the database robust trumps duplication.

-cranioscopical (February 14, 2020, 08:17 AM)

I agree.  I think it should all go into the database, but given that you (mouser) are worried about database bloat, a good backup and purge feature might help in that regard.  Perhaps a dropdown listing 1, 3, 6, 9, 12 month or older, and a button to backup and purge clips beyond the selected cleanup period.  On click, open a file dialog to the db directory and provide a default purge database name based on the selected cleanup period and the current date.  Maybe even give the option to just purge without backup (not the default) because there will probably be many who would be fine with a straight delete without save.

But now that I think of it, aren't there already automated backup, and purge settings?

[mouser]

New version uploaded.
I had to make some more elaborate changes to the multi-format "blob" data used by chs, such that previous blob data in clips from prior alpha versions won't be readable by this version.

I've still got quite a bit of work to do bringing everything in line with the new code, but CHS should be pretty good at saving and restoring the complete set of formats associated with clips, and hopefully handling duplicate detection, etc.

[mouser]

So what I've done so far:

• I've migrated the core clipboard capturing and pasting functions from using a single format (plaintext or image), to using a multi-format "blob" data structure.
• So when capturing it captures all formats in the clipboard and saves the collection (blob) of formats in the database.
• It has some basic options for what formats you want it to ignore, and tries to identify redundant formats that can be ignored, for example unicode/oemtext/dib bitmaps that are equivalent to other stored formats.
• It also handles plaintext and images specially; while preserving their order in the blob collection (if multiple formats found), it pulls out images to external png files with comments, and pulls out plaintext to sit in the database as a separate searchable field.  In this way, I'm hoping to get the best of both worlds (multi-format support but more convenient access to plaintext and images and minimizing database size).
• Order of the multiple formats in a blob is preserved even in the case of images and plaintext, which may occasionally be important to some scenarios.
• A single checkbox in the option will toggle the program from capturing multiple formats vs the old pure-plaintext behavior of CHS, so if you don't want the new formats and prefer a leaner pure plaintext database you can still use CHS that way.
• Duplicate checking should be somewhat improved, and work for multiple-format blobs and images.

What's left to be done:

• Lots more testing -- I'm hoping to get some help with that from the forum.
• There are a bunch of ancillary things regarding copying+pasting that I might not have fully implemented in the new code (flashing clipboard icon on capture, dealing with stubborn situations where the clipboard is being used by another app, updating clip modification dates on pasting, etc.).
• There are a few functions that are specifically plaintext related, like when performing format presets, and pasting text using sendkeys, that I have not fully moved over, or that use some old code.  I still have to go through and update all of those functions.  They will will only work with plaintext, but I think the existence of multi-format blob data may confuse some of these functions.
• Similar behavior might be expected after manually editing a clip -- you will have changed the PLAINTEXT content of the clip, but not the unicode/richtext version, So I think what has to happen in such cases is a clearing of the unicode/richtext data in such cases.
• I want to add more features to help you see how much space is being used by the multi-format data, and to do some maintenance operations on it to keep your database lean (like functions for removing multi-format data from old clips, etc.).
• I would like to add more flexible rules for deciding which formats you want saved.. For example being able to specify a custom list of formats to keep or ignore, and maybe setting arbitrary limits on size, or based on the application running.  I'm not sure the best way to do this that would satisfy experts and novices alike..
• Verify that the new image capture/pasting DIB format works in all cases.

[mouser]

Ok new version uploaded.. I'd like to get some feedback before I do too much more.
You might want to click the small button to the far left on the grid header row to show some of the columns to see more Blob info, etc.

[djMot]

I'm still not seeing images in a the HTML Preview tab.  Is there something I'm missing, or are you decisive about just not going there?

[mouser]

I'm still not seeing images in a the HTML Preview tab. 

-djMot (February 18, 2020, 08:21 AM)

Try re-downloading to get the latest version and then enable this option:

[mouser]

Here's a little trick, create a virtual folder(group) showing you only Large clips in your database (those bigger than 100k):



There are actually now 2 fields related to a clip's size.  DataSize is total size of clip INCLUDING any external image size.  BlobSize does not include size of external image.

You can click on the little button to the far left on the column header to see and show more clip field names.

[mouser]

Updated with some improvement to the organization of the options dialog.

[djMot]

I'm still not seeing images in a the HTML Preview tab. 

-djMot (February 18, 2020, 08:21 AM)

Try re-downloading to get the latest version and then enable this option:
[ Invalid Attachment ]

-mouser (February 18, 2020, 05:07 PM)

Yup!  Beautiful!  THANK YOU!   

[djMot]

Here's a little trick, create a virtual folder(group) showing you only Large clips in your database (those bigger than 100k):
[ Invalid Attachment ]


There are actually now 2 fields related to a clip's size.  DataSize is total size of clip INCLUDING any external image size.  BlobSize does not include size of external image.

You can click on the little button to the far left on the column header to see and show more clip field names.

-mouser (February 18, 2020, 09:25 PM)

Interesting.  So the SQL text box contains a WHERE clause, 'eh?  Is there a database schema somewhere so we can understand what's available to construct a WHERE clause from?

[mouser]

Is there a database schema somewhere so we can understand what's available to construct a WHERE clause from?

I will try to make one.  The help file needs some real updating.

[mouser]

Any more thoughts about what features are most important that still need to be added to make this multi-format stuff useful?

[tomos]

A quick look at the most recent version:

1. vector clips are being duplicated.

2. I havent used CHS for images of late. I changed default image size to 10,000 and copied a few images under 1MB in size (from dopus). The clips are listed but no images are showing in quick-paste nor in main window. If I copy an image again, I get the 'I'm already in the database' sound.  Am I doing something wrong?

Blob info

2.01kb (19 formats)

"DataObject" [0000C009]: 8b (query 0ms)
"Shell IDList Array" [0000C0FB]: 367b (query 0ms)
"dopus_cf_hdrop" [0000C2F1]: 90b (query 0ms)
"CF_HDROP" [0000000F]: 90b (query 0ms)
"FileName" [0000C006]: 34b (query 0ms)
"FileNameW" [0000C007]: 68b (query 0ms)
"dopus_cf_typetable" [0000C369]: 5b (query 0ms)
"dopus_cf_sizetable" [0000C36A]: 40b (query 0ms)
"DOpusSourcePath" [0000C365]: 40b (query 0ms)
"dopus_cf_rotation" [0000C367]: 2b (query 0ms)
"InShellDragLoop" [0000C10A]: 0b (query 0ms)
"Preferred DropEffect" [0000C0E3]: 4b (query 0ms)
"Performed DropEffect" [0000C106]: 4b (query 0ms)
"Paste Succeeded" [0000C108]: 4b (query 0ms)
"TargetCLSID" [0000C10C]: 16b (query 0ms)
"dopus_cf_sourcethread" [0000C36B]: 4b (query 0ms)
"dopus_cf_fromopus" [0000C36E]: 4b (query 0ms)
"dopus_cf_namespace" [0000C366]: 0b (query 0ms)
"Ole Private Data" [0000C013]: 936b (query 0ms)
-----

Extended (non-redundant) data save size: 2.01kb
Hash: 4D5AE5F1

3. HTML: it's very nice that the url is saved when copying text from a webpage

[mouser]

Can you give me some screenshots to show me what you are talking about regarding #1 and #2?

[mouser]

it's very nice that the url is saved when copying text from a webpage

Yeah i think i will add code to grab those source urls and put them into the normal clip info, so they are available in the grid and even if you don't store html.

[tomos]

Can you give me some screenshots to show me what you are talking about regarding #1 and #2?

-mouser (February 19, 2020, 07:25 PM)

I only have the ancient Freehand9 on this machine (but presume that that is not actually relevant). The vector clips from Freehand shown here are identical:

Clipboard Help+Spell Extended Clipboard Format Data Discussion Thread

1,2,4-9 and A are all image files copied from dopus, shown below (21=1, etc)

Clipboard Help+Spell Extended Clipboard Format Data Discussion Thread

[tomos]

The vector clips from Freehand shown here are identical:

-tomos (February 19, 2020, 08:05 PM)

FWIW the blob info has different hash numbers

[IainB]

@mouser:

Any more thoughts about what features are most important that still need to be added to make this multi-format stuff useful?

-mouser (February 19, 2020, 06:07 PM)

   

Yes, I have a couple:

• Enable wiki-like linking within Clip contents, to other CHS Clip IDs in the same database.
• Enable viewing and editing of RTF and HTML clip content in the Memo pane (similar to last ß version NoteFrog).

The changes you are making would seem to be in line with some already defined requirements, as detailed in User Requirements for CHS
I gather some other users have added their requirements to that document.
The more you could meet of those requirements, the better!   

[mouser]

The vector clips from Freehand shown here are identical:

Ok, so what you are saying is you copied something multiple times, and CHS is not figuring out that the clips are identical and should be regarded as duplicates?
I noticed this happening in firefox too, but as you point out, the hashes were different, so I'm guessing that their is something in the blob data that is NOT identical, like perhaps a timestamp.

The solution to this is:

• 1. Live with it and don't worry about having duplicate clips in such a case since it won't happen too often.
• 2. Wait for rules that would let you ignore blob items irrelevant to the content, which could solve your problem if the timestamp (or whatever) is not the main vector content.
• 3. I may add display of hash values for the individual format chunks in a blob so you could ID which one is changing..

I'll also try to add a better caption in the quick paste menu for blob data that has no text.
It might be nice to have a hex viewer dialog you could bring up for blob contents so you could actually see the raw contents of the blob data for this kind of debugging.

[mouser]

Updated with some fixes.  No more blank captions on quick-paste popup if a clip has no text or image.
Grabbing urls when copying from web pages is now done by looking at the html source content (and saved in Notes field of clips); in addition to being more robust it captures much faster and solves the slow firefox capture in previous versions.

I'd appreciate testing on more web browsers to see if there are any where CHS can't capture the urls. (i have tried firefox, ie, and chrome).

[tomos]

Ok, so what you are saying is you copied something multiple times, and CHS is not figuring out that the clips are identical and should be regarded as duplicates?
I noticed this happening in firefox too, but as you point out, the hashes were different, so I'm guessing that their is something in the blob data that is NOT identical, like perhaps a timestamp.

The solution to this is:

    1. Live with it and don't worry about having duplicate clips in such a case since it won't happen too often.
    2. Wait for rules that would let you ignore blob items irrelevant to the content, which could solve your problem if the timestamp (or whatever) is not the main vector content.
    3. I may add display of hash values for the individual format chunks in a blob so you could ID which one is changing..

-mouser (February 19, 2020, 08:54 PM)

not currently a problem for me (and probably wont be in the future), but for reference, here I compare two blobs -- more different than I thought...

Clipboard Help+Spell Extended Clipboard Format Data Discussion Thread

[tomos]

1,2,4-9 and A are all image files copied from dopus, shown below (21=1, etc)

-tomos (February 19, 2020, 08:05 PM)

can anyone comment about images: am I misunderstanding how that works? Can I not simply copy an image file?
@mouser, still nothing shown in most recent alpha for images as per screenshots in quoted post

[tomos]

[Request / rambling thoughts]

nice would be the ability to name image (and vector) clips. Could e.g. then be saved in favourites with clarity as to what exactly it is (often not clearly visible in thumbnail).

Can we actually *stop* image thumbnails showing in the quick-paste menu /sub-menus? Just curious -- I believe I would mostly reuse vector clips from the favourites and would be happy enough to not be able to see thumbnails if they have relevant titles.