Author Topic: General brainstorming for Note-taking software (Read 900215 times)

Nod5 · « **Reply #550 on:** June 14, 2007, 03:47 AM »

Darwin & others,
Adding an entire archive of PDFs to EndNote items manually must be extremely tiresome. Given the popularity of EndNote I bet many researchers are doing just that anyway (they probably curse a lot in the process

).

So if there's no tool to programmatically add the PDFs then maybe smart folks at DonationCoder could look into how hard it would be to make one? If such a tool could be completed it could draw a lot of interest to DonationCoder. And hey, it might due to saved time speed up scientific progress a (very, very, very small) bit.

The main problem to solve would be how to match the pdf files and the bibliographic items. If journal articles contain their DOI id number as extractable metadata then that might be a solution. I opened a few journal articles in UltraEdit and browsed for doi numbers or something similar in the ascii text but couldn't find anything. Also, I exported metadata from some sample PDF files with the free tool PDFTK. Here's one example of what I got:

InfoKey: Author
InfoValue: x
InfoKey: GTS_PDFXVersion
InfoValue: PDF/X-1:2001
InfoKey: Producer
InfoValue: Acrobat Distiller 6.0.1 for Macintosh
InfoKey: Creator
InfoValue: InDesign: pictwpstops filter 1.0
InfoKey: ModDate
InfoValue: D:20070613112031+01'00'
InfoKey: GTS_PDFXConformance
InfoValue: PDF/X-1a:2001
InfoKey: Title
InfoValue: 14.6 News 758-759.indd NS new.indd
InfoKey: CreationDate
InfoValue: D:20070612095515+01'00'
PdfID0: cee3243181236aabefbac383315fda1f
PdfID1: 3ab8e10f7074409da8a4fc9d6eb3a2
NumberOfPages: 2
PageLabelNewIndex: 1
PageLabelStart: 758
PageLabelNumStyle: DecimalArabicNumerals

this row "InfoValue: 14.6 News 758-759.indd NS new.indd" is sort of an identification since the PDF is from Nature Volume 447 Number 7146 section News page 758. But it's not as good as a DOI. And other articles i tried did not even have that InfoValue. PdfID0 & PdfID1 are made by Acrobat Distiller I think, and does not relate to the article contents/DOI.

Embedding DOI numbers according to some standard seems like an obviously smart thing. If available it would not only allow adding a pdf to the corrent EndNote items but also the reverse. Start with a pdf, autoresolve its DOI online, then import all the resolved metadata into an EndNote item, rename the pdf according to some format and import/link it to the EndNote item.

But it wouldn't really surprise me if DOI is not embedded at all. The electronic journal systems still seems rather ineffective and user-unfriendly. As this overview states (http://hublog.hubmed...archives/001306.html ):

While most of the larger publishers provided an acceptable method of authentication, the PDF files they produce are obviously not optimised for ease of use by the reader. It's almost impossible to build a tool to automatically fetch PDFs for papers [---] The implementation of all of these features could be automated with little change to the publishers' systems, but would be a major benefit to researchers struggling to deal with large amounts of literature.

(BTW, a comment on that page links to http://quosa.com/solutions.html , a tool for easy, massive article downloading and EndNote importing. No price is listed on the site (only this note on discounts http://www.quosa.org...docs/mac/pricing.htm ). So I suspect it is very expensive. )

Without DOI in the PDFs any programmatic matching of PDF and EndNote items would probably have to be more fuzzy.

nevf · « **Reply #551 on:** June 14, 2007, 07:42 AM »

I think I can address some of what you requested using UltraRecall (http://www.kinook.com). Here is how...

UR has a feature that I have not seen in any other PIM system. It allows you to link (or store) any document on your system e.g. pdf, doc, xls, OL items, anything. You can actually store the doc within UR database and delete it from the OS.
-cnewtonne (June 06, 2007, 10:32 PM)

I just want to let everyone know that Surfulater also does this. ie. Store any files in its Knowledgebases or add links to them. Then open them in their native app. You can drag and drop files from Explorer on to an Articles Attachments field or use the context menu to add links or attach files. Synchronized editing of embedded (attached files) is planned. Files with links are of course always up to date.

Surfulater version 2.00.30.0 has been released today. Download from http://www.surfulater.com

Darwin · « **Reply #552 on:** June 14, 2007, 10:07 AM »

PDF import into Endnote is only cumbersome if you have several thousand to do "after the fact". What you do is tile an endnote window and a file manager side by side. Then you just drag the pdf onto the title of its entry Endnote and it is automatically copied to the Endnote managed folder and a link created to it. It's not too cumbersome if you do it as you download/save pdfs. Of course, it would be preferable, if you are having Endnote manage the pdfs to save the pdfs directly to Endnote, but I've yet to see that as an option. Alternatively, you can simply have a link created that points to wherever you happen to have the pdf stored. This functionality was introduced several versions back... As I noted above, if I had had my head screwed on right I would have realised that having Endnote store the pdf's is silly given the size of even a small library of pdfs and that v.9 already did all that I needed...

The soft-link to your pdf's is preferable because it allows you to organise your pdfs - I had mine very organised - my folder structure goes: PDF - Journal - Year and in some cases Volume. All pdf's were saved with file names that reflect the title of the work (if I was starting again I'd put author name first, year of publication and then the title, but can't go back 7 years, sadly. Endnote just dumps the pdfs into it's own folder in a common folder under the library folder. So, Endnote libraries - Library Name.data - PDF. Fortunately, desktop search technology and/or Endnote's own system mean that nothing is far from "hand". I have found this sort of liberating because occasionally I'll just save the pdf with its default name (science.pdf or article.pdf or 012387.pdf) and don't have to worry about finding it again.

Now, to get to NoD5's point finally (if I am getting it) - it would be great to have an app like Endnote scan a pdf and extract the information from it to generate the Endnote entry. I know that this would SERIOUSLY increase the size, complexity and footprint of Endnote, but it would be useful. If free search agents can scan a pdf in under a second and catalogue its content, surely an app could extract data from it to insert into a database. Of course, this would require some sort of standardised formatting on the part of publishers and authors OR the creation of some sort tag that contains the info (a la mp3s). Fortunately, most publications now have a "download citation" feature and will auto insert everything into a new Endnote library entry. Sweet. Now if we could just save a pdf directly to Endnote...

Surfulater looks interesting - this kind of functionality is frequently requested for TexNotes/Do-Organizer, but it's been pointed out that this would significantly increase the size of the database. The developers have promised to consider it but so far nothing as come of it (at least with TexNotes - I've not really been following Do-Organizer's development). Anyway, my point being that this would be a useful feature. The issue is with portability - if one is not concerned about that, then it's not an issue.

superboyac · « **Reply #553 on:** June 14, 2007, 10:19 AM »

One question about Endnote users:
Would you use Endnote for keeping track of your pdf's and documents if you weren't using it for research? Like, if you didn't already have Endnote for school purposes (or some other academic application) you wouldn't go out and buy it for it's ability to annotate your documents, right?

I'm just trying to get a feel who is using Endnote this way. In my case, I'd probably go with Surfulater or something a little less expensive, especially if I didn't it for a properly formatted bibliography.

urlwolf · « **Reply #554 on:** June 14, 2007, 12:38 PM »

Darwin & others,
Adding an entire archive of PDFs to EndNote items manually must be extremely tiresome. Given the popularity of EndNote I bet many researchers are doing just that anyway (they probably curse a lot in the process ).
-Nod5 (June 14, 2007, 03:47 AM)

You can bet on that.

Embedding DOI numbers according to some standard seems like an obviously smart thing. If available it would not only allow adding a pdf to the corrent EndNote items but also the reverse. Start with a pdf, autoresolve its DOI online, then import all the resolved metadata into an EndNote item, rename the pdf according to some format and import/link it to the EndNote item.

But it wouldn't really surprise me if DOI is not embedded at all. The electronic journal systems still seems rather ineffective and user-unfriendly.

Shameless plug: I have written about that here:
http://www.academicp...ng-around-with-pdfs/

As much of a pain as it is to manage a music collection (tags are always incomplete/wrong) a pdf collection is much worse.

A possible solution would be to implement fingerprinting for pdf (like what musicIP does) and then tag these pdfs with at least author, year, etc. Looks like the pdf specs support some tagging:

from atom prober in the comments:
PDF supports XMP. XMP allows all the dublin core metadata that Zotero, refbase, OpenOffice.org, and other products are using.

We just need to have publishers care enough to put this data in and more end-user tools to index/view/search/edit it.

Now we only need someone to code/maintain a central repository of pdf metadata, and mappings fingerprints -> ids. Sooner or later, publishers would start incorporating metadata as well.

You know, this is not a bad idea. How many people would use it?
Then, zotero, endnote, etc could work like any mp3 tagger/player.

That'd save lots of headaches!

Darwin · « **Reply #555 on:** June 14, 2007, 01:22 PM »

One question about Endnote users:
Would you use Endnote for keeping track of your pdf's and documents if you weren't using it for research? Like, if you didn't already have Endnote for school purposes (or some other academic application) you wouldn't go out and buy it for it's ability to annotate your documents, right?

I'm just trying to get a feel who is using Endnote this way. In my case, I'd probably go with Surfulater or something a little less expensive, especially if I didn't it for a properly formatted bibliography.
-superboyac (June 14, 2007, 10:19 AM)

No, you're quite right. If I wasn't using it for academic purposes I probably wouldn't even consider Endnote. Which is not to say that I think it's bad and not to say that I'm going to dump it when (if) the PhD (ever) gets finished.

Nod5 · « **Reply #556 on:** June 14, 2007, 03:33 PM »

urlwolf,

A possible solution would be to implement fingerprinting for pdf (like what musicIP does) [---] Now we only need someone to code/maintain a central repository of pdf metadata, and mappings fingerprints -> ids.

This is a great idea that completely bypasses the need for DOI extraction. One way I can see it happening would be if some popular application like Zotero implemented this as an opt in feature that works automatic in the background. That is, every time someone downloads both article metadata and a pdf through Zotero, Zotero silently uploads pdf fingerprint and matching metadata to some server. As the database grows, downloading just pdf files will be enough since the metadata is already available in the open archive. Zotero seems like the kind of tool that is innovative and community driven enough to be ready to pioneer something like that.

One problem might be related to protected metadata. Some journals require a login just to see the abstracts for instance. So I'm not sure about the legality of storing such abstracts in an alternative, open archive. One way around that would then be to match and archive only pdf fingerprints and DOI numbers and then let Zotero and so on implement some way to later automatically resolve the doi and grab metadata from the resolved article page (including abstract if the user is authenticated to have that displayed). Another advantage with such piggybacking on the DOI system is that that the archive then never risks having outdated article links. Another problem is if journal publishers change the pdf files from time to time. But perhaps that can be solved just by letting the archive match the multiple fingerprints to the same metadata.

edit: great post on Academic Productivity also

Darwin · « **Reply #557 on:** June 14, 2007, 05:36 PM »

urlwolf - I wrote a lengthy reply here AND then went and read your academic productivity blog on the subject. Wow! All I can say is: "what he said". You covered all the bases, well done!

Nod5 · « **Reply #558 on:** June 19, 2007, 07:41 AM »

This might be useable for pdf notetaking and archiving: Adobe Digital Editions. Will try it out later myself.
http://www.adobe.com...cts/digitaleditions/

http://www.adobe.com...digitaleditions/faq/
What is Adobe Digital Editions? Adobe Digital Editions is a new, exciting rich Internet application (RIA) built from the ground up for digital publishing. Digital Editions is a simplified, engaging way to acquire, manage, and consume eBooks and other kinds of digital publications.

Will a Linux version of Digital Editions be available? Yes, a desktop Linux is under development and a public Beta is expected later this year.

What file formats does Digital Editions support? PDF/A and an OPS (enhanced OEBPS, which is a profile of XHTML 1.1 with OCF packaging) are the two publication-level content types. SWF content and common image file types can be referenced within PDF and XHTML-based publications.

What are the Digital Editions annotations capabilities? Digital Editions 1.0 supports bookmarks, highlights, and text notes via its "bookmarks" panel. These annotations are stored in an open XML format separately from publications, in order to enable seamless annotating across PDF and XHTML-based (EPUB) publications, and to set the stage for future social networking features (e.g. sharing annotations within a community of readers).

Darwin · « **Reply #559 on:** June 19, 2007, 09:04 AM »

Interesting find NoD5 - I'll check it out, thanks!

Armando · « **Reply #560 on:** July 08, 2007, 06:07 PM »

Have decided to try Zoot, for the fun of it. Unfortunately I get this error after about 10 min. :

"The Win 16 Subsystem has insufficient resources to continue running. Click on OK, dose your applications, and restart your machine"

Never had that before with any other programs. I must admit that I don't use any 16-bit app.

Oh well.

alxwz · « **Reply #561 on:** July 08, 2007, 08:24 PM »

There should be a beta of Zoot v. 5 (32 bit) coming these days, at last.

However, I did register for the beta testers list a while ago, but didn't get any notice up to now.

Armando · « **Reply #562 on:** July 09, 2007, 06:03 PM »

There should be a beta of Zoot v. 5 (32 bit) coming these days, at last.
-alxwz (July 08, 2007, 08:24 PM)

Nice. I guess I'll just wait then

However, I did register for the beta testers list a while ago, but didn't get any notice up to now.

There seem to be a great deal of suspens around the development of Zoot 5...

alxwz · « **Reply #563 on:** July 09, 2007, 06:26 PM »

Sorry I didn't put this precisely enough.
This is most likely a non-public beta and I'm not exactly sure who qualifies as a beta tester.
Please see the original announcement (May, 31) by Tom Davis on Yahoo groups:
http://tech.groups.y...tForum/message/11696
and this post from June, 21
http://tech.groups.y...tForum/message/11726
and the ongoing discussion.

Armando · « **Reply #564 on:** July 10, 2007, 03:38 PM »

I like locate also. And for searching inside documents, archivarius. Combine that with a good web capturing and notetaking tool like Evernote, or surfulater, or something else, and you have a pretty good overall system.

-superboyac (June 13, 2007, 12:00 PM)

Hi superboyac. Buzy these days ?

I don't want to duplicate posts so here's a link to what I've posted in that other thread. I believe that it echoes with what you said (and to some of what has been said here too, in this huge thread. Nothing new maybe…). I too am waiting for what zoot 32 bits is going to bring to the table...

Edit : Oups! Forgot the link

https://www.donation...69.msg67645#msg67645

Armando · « **Reply #565 on:** July 10, 2007, 03:38 PM »

Sorry I didn't put this precisely enough.
This is most likely a non-public beta and I'm not exactly sure who qualifies as a beta tester.
Please see the original announcement (May, 31) by Tom Davis on Yahoo groups:
http://tech.groups.y...tForum/message/11696
and this post from June, 21
http://tech.groups.y...tForum/message/11726
and the ongoing discussion.
-alxwz (July 09, 2007, 06:26 PM)

Thanks for the info!

Armando · « **Reply #566 on:** July 10, 2007, 10:36 PM »

Maybe something should be said about EverNote v2.1 to those who... haven’t upgraded on June 25th?

First, see this thread : https://www.donation...59.msg66740#msg66740
... Then look at these nice changes :

Public Release 2.1.0.327 (June 25, 2007)
New Features:

NEW NAVIGATION FUNCTIONALITY
•   Ability to sort the Note List and underlying tape notes by Note List's columns significantly simplifies navigation.
•   Category Intersection Panel can be opened from the View Menu.
•   New shortcuts have been added for Categories Window, Note Lock, Delete Note for keyboard-oriented users.

IMPROVED EDITING
•   Each note can be converted into a To-Do item with a new checkbox available in Note Bar.
•   Basic image editing in the notes has been added - a new right-click menu item "Edit Picture" provides various image editing options.

ENHANCED CLIPPING
•   The clipped content can be instantly sent via email using a new button in the Notification window for all EverNote clippers including Universal Clipper.

NEW UNIVERSAL CLIPPER FUNCTIONALITY
•   Ability to add ink markup to image has been added. This feature allows writing over the selected screen area and then saving the annotated image to the EverNote tape or as a separate file.•   EverNote application will be launched automatically during clipping even if it was not running.
•   Ability to specify target database has been added. It can be done either inside of the My.EverNote.com context menu or while clipping by holding Shift.
•   The Universal Clipper now determines whether EverNote is busy at the moment of clipping and sends an appropriate message (instead of previous "Can't clip" error message).
•   The highlighted screen area can be saved using the Ctrl+S shortcut.

NEW EXPORT AND IMPORT FUNCTIONALITY
•   New Import Wizard sets up automatic addition to the tape of the content from any folder and its subfolders, including folders on feature and camera phones connected via USB port.
•   Auto-import for POP3 emails has been added. It allows importing snapshots sent from camera phones.

MISCELLANEOUS
•   Silent installation mode for EverNote Clippers for Mozilla Firefox and Thunderbird has been added. Installations will be done for "current user" or for "all users" depending on the selection for the whole setup.
Bug Fixes & Improvements:

LOOK AND FEEL
•   Short/full note height is now saved in the note.
•   Screen location of the Notification Window is now saved in registry.

CATEGORIES
•   The ability to create sub-categories with the same name in different hierarchies has been restored.
•   Fixed a bug that prevented dropping keywords onto categories.
•   Fixed a bug that prevented a note from becoming "Uncategorized" after all assigned categories have been deleted.

EDITING
•   Fixed a bug that caused incorrect handling of the Ctrl-Alt-V shortcut in the full-screen editing mode.

NAVIGATION
•   Fixed mouse-wheel scrolling issues with IntelliMouse driver for Windows Vista.

EXPORT, IMPORT, AND BACKUP
•   Fixed a bug that caused Auto-import memory leaks.
•   Fixed a bug that caused EN crashes during database restore operation if no database is open.
•   The backup file name origins from the database name instead of previously used numeric combinations.
•   Fixed a bug that prevented restoring a database from a "read-only" backup file.

ADVANCED NOTES RECOGNITION (ANR) FOR INK NOTES
•   Increased recognition accuracy by using more advanced algorithms.
•   Implemented automatic title generation for ink notes (in EverNote Plus only).
•   Fixed a bug that caused interruption of ink collection during auto-saving of the database.
•   Fixed a bug that caused EverNote crash after a long ink note has been created.

CLIPPING

All clippers
•   All EverNote Clippers are integrated with EverNote Outlook Extension.

EverNote Universal Clipper
•   Fixed a bug that caused unblocking Win+A while the Resize Selection dialog box is open.
•   Fixed a bug that caused Universal Clipper to crash if selection area was extended beyond the screen area using keyboard.
•   Fixed a bug that caused Clipper to always start minimized.
•   Fixed a bug which caused memory leaking after each insertion of captured image into the EverNote tape.
•   Fixed a bug that caused improper positioning of settings window in a multi-monitor mode.

EverNote Clipper for Microsoft Outlook
•   Clipper now adds "Subject" line in the clipped email.
•   Ensured that a clipped portion of email is stamped by clipping time rather that the time of email receipt.
•   Fixed a bug that prevented Outlook Clipper from clipping into a database opened in EverNote Outlook Extension.

EverNote Web Clippers for Mozilla Firefox and Thunderbird
•   Fixed a bug that distorted clipped messages from Thunderbird.
•   Fixed a bug that assigned different titles to clips of whole mail and its portion.

INSTALLATION
•   The setup suggests to close Mozilla Firefox/Thunderbird before it can install/update/uninstall the EverNote Clippers for those applications.
•   Fixed a bug that prevented EverNote from starting under a limited user profile after a successful installation.
•   The setup checks disk existence if the media was changed during the installation.

MISCELLANEOUS
•   Help file has been updated for this release.
•   The seed database has been updated - all broken links in sample/welcome notes have been fixed and orphaned categories have been removed. Text Notes category has been removed.
•   Updated Demo Flash Movie - fixed typos and date format.
For more information also see the Known Problems page.
________________________________________

http://www.evernote....ote/releasenotes.php

I've put in red the bits that immediatly affected (or will affect) positively my work. The "ability to sort the Note List and underlying tape notes by Note List's columns" is probably the greatest addition. (Please note : I don't work for EverNote

)

Now… if only the next version could ...
- allow opening several windows of the same database
- have some kind of "commenting tool" / higlighter.
- allow to dellete/edit several categories at once
- allow adding multiple filters (properties --> filter tab ; right now you can only select one filter -- keywords OR history OR templates...)
I suggested that in the forum, but they didn't seem to think it was that important... Oh well.

Armando · « **Reply #567 on:** July 13, 2007, 02:08 AM »

Yet another "note-taking" software -- seems to be more in the "web-clipping and surfulater family", if I may say so...

http://www.macropool...fessional/index.html

Here's a review

http://www.outliners.../0/web-research--wow

twinkler · « **Reply #568 on:** July 15, 2007, 04:36 PM »

http://www.outliners.../0/web-research--wow
-Armando (July 13, 2007, 02:08 AM)

Folks might be interested in taking a look at that site, http://www.outlinersoftware.com. It's a discussion forum for people "interested in outlining, PIM, and knowledge management software."

DBC · « **Reply #569 on:** July 27, 2007, 10:27 AM »

Folks might be interested in taking a look at that site, http://www.outlinersoftware.com. It's a discussion forum for people "interested in outlining, PIM, and knowledge management software."
-twinkler (July 15, 2007, 04:36 PM)

I'm sure folks over there would be very happy to see you all!

"Outlinersoftware.com" was originally "outliners.com/discuss," a forum set up by Dave Winer in the 90s to exchange views about outliners (mainly single-pane ones). Nowadays, as well as outlining, it touches on other aspects of researching and writing as well - web clipping, pims, note-taking, editing, word-processing, etc. It has a big searchable archive going back many years. The move to a new server somewhat disrupted the little community, but it is recovering well.

DBC

Armando · « **Reply #570 on:** July 27, 2007, 01:13 PM »

Folks might be interested in taking a look at that site, http://www.outlinersoftware.com. It's a discussion forum for people "interested in outlining, PIM, and knowledge management software."
-twinkler (July 15, 2007, 04:36 PM)

I'm sure folks over there would be very happy to see you all!

"Outlinersoftware.com" was originally "outliners.com/discuss," a forum set up by Dave Winer in the 90s to exchange views about outliners (mainly single-pane ones). Nowadays, as well as outlining, it touches on other aspects of researching and writing as well - web clipping, pims, note-taking, editing, word-processing, etc. It has a big searchable archive going back many years. The move to a new server somewhat disrupted the little community, but it is recovering well.

DBC
-DBC (July 27, 2007, 10:27 AM)

I've read lots of stuff there and really enjoyed the info!
Some developers post there too (Ultra Recall developper, if I remember well?), which adds some spices to the sauce.

iphigenie · « **Reply #571 on:** July 28, 2007, 10:17 AM »

I'm starting to like evernote

In a way it is a path i am on which relies less on my having to impose structure, and more on tag and search.
I went from a traditional folder based mail software to using opera m2 (private) and Outlook with everything in the same folder and use of tagging / filtering / searching to create a virtual organisation
I went from organising my images in topical folders to storing them by year/month/day and using tagging and metadata
And now I might go from using an outliner type product to one where the structure is all tags

seems to all fit

rjbull · « **Reply #572 on:** July 30, 2007, 04:13 AM »

Interesting thread here: Storing Nuggets of Information
both paper and computer methods

superboyac · « **Reply #573 on:** July 30, 2007, 11:50 AM »

Well, I finally got a chance to beta test the new 32-bit version of Zoot. Not much has changed besides the behind-the-scenes conversion from 16-bit to 32-bit. This is going to be the standard bearer for notetaking in my opinion, I'm glad it's finally being resurrected.

I know my notes have become too unwieldy to maintain in a tree hierarchy, so some other form of organization will have to replace it. Zoot's organization features are so powerful, no other program offers it. It's automatic rules, filtering, categorizing, etc is really great (I would liken it to a really powerful email filtering system). Evernote is doing a decent job in this area, but once Zoot is finished in the 32-bit version, Evernote will just be eye-candy.

tomos · « **Reply #574 on:** July 30, 2007, 12:33 PM »

I'm starting to like evernote

In a way it is a path i am on which relies less on my having to impose structure, and more on tag and search.
-iphigenie (July 28, 2007, 10:17 AM)

I'm finally getting into using evernote properly, I like it a lot too but find the lack of shortcuts WRT categories a bit of a pain - maybe it's especially at the beginning when you need to continually set up new categories & maybe modify keywords for older ones etc. and it's all mouse clicking...
BTW - do you know:
is there a way to get it to ignore a keyword if it is only part of another word - eg keyword monitor brings up monitoring which i dont want ?