Automatic document creation. How?

[superboyac]

My business creates practice tests.  Right now, it is all being done manually using Indesign.  It is VERY manual.  Text boxes are placed manually, graphics are imported and placed manually.  Labels and symbols are tweaked on an individual basis.  So I'm looking for a way to automate this entire process.  I feel like it should be possible because the structure of the pages is very consistent.  There are really just a few elements per problem:
--Question
--Answer Choices
--Supplemental diagrams/graphics
--Solution

That's it.  So I can very easily store all of this information in a database.  Then, I can somehow have the database randomize the order and selection of a test to generate.  Boom...one-click test generation!  So that's my vision.

But how do I go about doing this?  All of this desktop publishing and database utilities are so mired in their corporate jargon that it's impossible to really know what these programs do.  It's extremely annoying.  I can read for ages and ages, and I'm still left thinking, "Just tell me what the FUCK this program does!!"

So here are my ideas so far, please comment on them if you have any knowledge about it:
Indesign Server and Incopy
Supposedly this is supposed to do what I'm talking about.  But hell if I know from what I read.  Lots of useless jargon.  And not enough screenshots, or examples, or any kind of customer testimonial.  Everything out there is just marketing jargon.  So I have how this beast works.  What I'm really concerned about is how easy it is to use.  I'm not afraid of a little scripting and such, but I don't want to spend a year just to get to the point to be able to play around with the program's features.  I would love to talk to someone who has used these two programs.

MS Access
This is my oddball solution.  Access is a database program, of course.  And it can do reports.  So I'm thinking, why don't I stick all the elements I mentioned above in an access database, and use the reporting feature to generate the actual test layouts.  I don't see why this can't work.  The other reason why I like this is because I find Access relatively easy to use for a database/programming type of application.  The problem is that using it in this way is very 'weird" to people and if I ask them about it, they will just wonder why the hell I'm doing this instead of using Word or Indesign or something traditional like that.  But I don't understand why this can't work, and work well.

A hybrid of software and databases
I'll just lump all the other alternatives here which are unknown to me at this point.  I'm sure I can use a mixed bag of sql databases with some other type of reporting application (Crystal Reports?, Visual Studio stuff).  Or some kind of workflow involving Access, Excel, Word, etc.  I have no idea.

As with most software, I want to use the one that is easy.  I don't think what I'm doing is terribly complicated, but because I want control and I'm picky, I can see why it can get complex.  As much as possible, I want to stay away from hardcore programming and scripting.  I don't want to be spending a lot of time learning java, xml, .net, c#.  This is why the Access solution appeals to me.  It strikes a good balance between power and user interface elements.

[Renegade]

I'd shy away from Access.

But you can do that any number of ways. XML + XSLT is one. Database => business logic (to create random set of questions) => XML & XSLT.

Or PHP or ASP with HTML & CSS templates. (Or any other scripting language.)

Those are pretty cheap and fast.

I don't know how you'd do that for InDesign though. I suppose there's a way to connect it to a database and use a template. InDesign is much nicer than HTML for presentation. You could automatically generate the basics, and pass it off for further design/tweaks.

[Stoic Joker]

I'd shy away from Access.

I think you mean run away shreaking in terror, right?

-------------------------------------------------

But seriously. SB How are the test being presented/taken now? Do you want a web interface with a question pool db backend? I'm actually not entirely sure what InDesign is/does ... So I'm trying to get my head around how you'd make a test with it. Are these printed?

I've had great luck doing a ton of projects based around MySQL. It's easy to work with and you can access it pretty much any way you like.

[JavaJones]

Have you taken a look at Framemaker? I do believe it's built for stuff like this.

http://www.adobe.com/products/framemaker/features.html

Powerful, scalable single sourcing. Effortlessly single source variations of the same document for different channels and purposes. Build and use complex Boolean expressions to define output filters in topic-based, structured, and unstructured modes.

Automation through scripting. Easily automate time-consuming, repetitive tasks through advanced scripting support with Adobe® ExtendScript. Run scripts for simple tasks from within Adobe® FrameMaker® or through the ExtendScript Toolkit to eliminate manual effort and increase productivity.

Structured Application Creation Wizard. Benefit from a basic infrastructure for working with structured Adobe® FrameMaker®. With this highly intuitive, UI-based tool, users can start working with structured FrameMaker even if they don’t have any prior knowledge or training.

Of course it's complex and would take time to learn and figure out how to apply it to your needs, but any really effective solution is going to be like that.

- Oshyan

[40hz]

There's a category of software called 'test generators' that will do some, but probably not everything you're trying to accomplish here. Their primary function is to generate random tests based on your selection criteria (random, weighed by subject, difficulty, etc.) from a pool of questions. Most have export options to word, PDF, and other formats. They're fairly utilitarian - so the output, while very legible, may not be up to what you're looking for if you're using InDesign. Most text publishers get around this by doing a formal "pretty typography" textbook and then include a less polished looking test booklet. Most customers for test review products don't seem to mind.

One program that's popular is PrimeExam. I had a client who did corporate training that used it. It seemed to get the job done.

A Google search for "test generator" will net you a selection of similar programs to explore.

Framemaker is an excellent program for creating large structured documents as JJ mentioned above. It's similar to InDesign, but the underlying product philosophy and workflow methodology is different enough that there will be a learning curve. Framemaker isn't a particularly difficult program to use. But it's also not a program that's easy to use correctly and effectively unless you put the time in to understand what it's about.

[JavaJones]

Yeah, I was assuming here that, due to the use of InDesign, the aesthetic quality of the output is important to some degree. Otherwise I'd just recommend a good automated test generator dealy, of which there are many.

- Oshyan

[Target]

difficult to say without knowing what you have in the way of source files, and or how much (and could be I'm talking through my hat here), but i suspect you could set something up in word, ie use a macro to merge randomised text into a template...

[superboyac]

There's a category of software called 'test generators' that will do some, but probably not everything you're trying to accomplish here. Their primary function is to generate random tests based on your selection criteria (random, weighed by subject, difficulty, etc.) from a pool of questions. Most have export options to word, PDF, and other formats. They're fairly utilitarian - so the output, while very legible, may not be up to what you're looking for if you're using InDesign. Most text publishers get around this by doing a formal "pretty typography" textbook and then include a less polished looking test booklet. Most customers for test review products don't seem to mind.

One program that's popular is PrimeExam. I had a client who did corporate training that used it. It seemed to get the job done.

A Google search for "test generator" will net you a selection of similar programs to explore.

Framemaker is an excellent program for creating large structured documents as JJ mentioned above. It's similar to InDesign, but the underlying product philosophy and workflow methodology is different enough that there will be a learning curve. Framemaker isn't a particularly difficult program to use. But it's also not a program that's easy to use correctly and effectively unless you put the time in to understand what it's about.

Boom! Thanks.  I just read a little about Framemaker and it sounds like it is meant for what I have described above.  I love the idea of "elements" and structured documents.  That's exactly what I want.  I'm going to play around with it.  Cross my fingers!  (I may have to learn xml a little bit, oh well, I've been avoiding it for years anyway, might as well jump in eventually)

[superboyac]

Yeah, I was assuming here that, due to the use of InDesign, the aesthetic quality of the output is important to some degree. Otherwise I'd just recommend a good automated test generator dealy, of which there are many.

- Oshyan

Yes, you are correct.  We are very picky with the presentation of our printed material.  That being said, it's not a terribly complex layout.  It can very accurately be considered a technical document, I just don't want it to look like an ultra boring journal-style layout (Latex anyone?).  We've chosen our fonts carefully, I have a discussion here about Helvetica and stuff.

I find this whole market of desktop publishing confusing.  And from my experiences, I can attribute it mostly to marketing jargon.  Like, it doesn't make sense to me that a certain program is good with typography.  What the heck does that mean?  To me, it means "stuff with fonts".  Why one program is better at it than another is a little confusing.  What is there to do with fonts?  They have a size, color, weight, character spacing (kerning), etc.  And all the programs can do that, even Word.

I get so frustrated reading about any software these days other than the cheapy stuff like $20 shareware.  For some reason, all the cheap stuff have no problem explaining exactly WHAT THE PROGRAM DOES.  Anything more than $100 has a very hard time telling me, in a concise manner, what the program actually does.  Here's my favorite example, Primavera:
http://www.oracle.com/us/primavera/index.html

Oracle has acquired Primavera Software, Inc., a leading provider of Project Portfolio Management (PPM) solutions for project-intensive industries.

Primavera offers best-in-class solutions focused on the mission critical PPM requirements of key vertical industries including engineering and construction, public sector, aerospace and defense, utilities, oil and gas, manufacturing and high tech, and IT and services.

Primavera's PPM products, together with Oracle's project financials, human resources, supply chain management, product lifecycle management, business intelligence, and infrastructure software are expected to provide the first, comprehensive Enterprise Project Portfolio Management solution. This solution is expected to help companies optimize resources and the supply chain, reduce costs, manage changes, meet delivery dates, and ultimately make better decisions, all by using real-time data.

Primavera is primarily known as the ultimate SCHEDULING program.  The word "schedule" doesn't appear anywhere on that page.  It's nonsense.  It's all marketing jargon.  It drives me out of my mind.

[Renegade]

Primavera is primarily known as the ultimate SCHEDULING program.  The word "schedule" doesn't appear anywhere on that page.  It's nonsense.  It's all marketing jargon.  It drives me out of my mind.

+1
Yeah... The ability to communicate is pretty rare.

[Target]

Primavera is primarily known as the ultimate SCHEDULING program.  The word "schedule" doesn't appear anywhere on that page.  It's nonsense.  It's all marketing jargon.  It drives me out of my mind.

+1
Yeah... The ability to communicate is pretty rare.

that's because it's written by world leading experts...

[superboyac]

Primavera is primarily known as the ultimate SCHEDULING program.  The word "schedule" doesn't appear anywhere on that page.  It's nonsense.  It's all marketing jargon.  It drives me out of my mind.

+1
Yeah... The ability to communicate is pretty rare.

that's because it's written by world leading experts...

nicely done!

[JohnFredC]

I spent 15 years of my professional life making my entire living as an independent Access developer so please bear with me.

No offense intended to anyone here, but those who poo-poo Access probably haven't used it professionally.  EDP staffs hate Access because skilled developers can create in 6 weeks what they quote management will take up to a year or more in SQL Server/Oracle (but only if management will approve additional staff   ).

If you need: password sign-in, boiler plate texts and objects, input validations, progress/completion management, score tracking, custom user interface that looks like a unique application, perhaps a kiosk mode, reporting and statistics, simultaneous mixed data stores, export to other tools... then Access can be your "man".  And if your deal is web, an Access application can even auto-generate boiler plate web page components and post websites (though not my personal expertise).

Some of my sites have had 15 simultaneous users on a shared Access backend on a server w/o problems.  Other sites use SQL and Oracle back-ends.  

Users don't need to know they are using Access:



If your deal is to collect, use, and report data, esp. in the business workplace, don't discount it for your project without a thorough look-see.

[superboyac]

JohnFredC, thanks for the input!
As an experienced Access developer, do you think my initial idea was a good one?  If I input all the elements of the book into Access: questions, answers, solutions, diagrams...as tables and all that, can I then create a report which would look like the sample that is shown on our website?  I don't see why not, but every time I ask someone who I think knows Access, they look at me like "Why the hell would you do it that way?"...anyway, please take a look at the sample of our book below, and I'd appreciate any comments:
http://www.compleximaginary.com/

[JohnFredC]

2superboyac

do you think my initial idea was a good one?

Indeed I do! 

If I re-state the obvious in the following, please pardon me.  But the obvious is often forgotten and IMO deserves restatement ad naseum.

After turning pages, perusing your book in the viewer at that link (assuming here that the page-turning viewer itself is not part of your project) I can say unequivocally: yes you can use Access to reproduce the book as either an interactive user interface for a person to take the exam with, or as a published document (that is, on paper), or any other output you conceive.  The beauty of Access is that it is ONE tool that can do all of what you need, efficiently.

But understanding what you need is crucial.

If I input all the elements of the book into Access: questions, answers, solutions, diagrams...as tables and all that, can I then create a report which would look like the sample that is shown on our website?

Yep.

The design, implementation, and distribution of the book is a relatively trivial task.  Step back a bit from the presentation of the book itself (though obviously that is your end-product) and observe that the book (design/layout, publishing channel, what-have-you) is not the real asset, but only derivative from the real asset. 

Instead, as you apparently already understand, the book (however it is distributed: print or electronic) is actually just one view (of many possible views) into a "sea" of data.  That data is the texts, the rules, the diagrams, the organization, the answers, the formatting, etc. This data is your asset, not the printed book.  All processes and systems associated with collecting, maintaining, and expressing this mass of data are costs.  Your goal should be to minimize these costs while at the same time leveraging the asset (the question data) into future opportunities.

What you need are routine tools and processes for collecting, managing, editing, connecting, presenting this data.  Address these tasks first, then you can slice and dice the data into a book (or books), as a website, as executable tests, as anything your little heart desires.

An application (or applications) for managing the data behind your book (oh and incidentally, preparing the book itself) sounds like a fun, straight-forward, Access project.  Access supports solutions that could run on a single disconnected workstation of minimal spec, to a full-blown groupware environment with shared data on a server... or both at the same time.  No other tools or expertise (beyond Access and VBA) needed.  Very cost effective and, might I add, future-proof, because you can always migrate the data, process, and gui to some other system, either all at once, or piece-meal.

A caveat is that such applications and data structures need to be properly designed to work as effectively as I describe.  But that is a requirement of any solution environment, not just Access.  The systematic structures of data and tasks are independent of the tools you use to instance a solution.

Who is your user?

[Jabberwock]

I see two problems here: first of all, all the described solutions could more or less do what you want. If you knew any of them, that would be exactly the easiest solution. It is rather difficult to say which of them would be easiest when starting from scratch...

The other problem is that the requirements must be specified very precisely. Typography is more than "stuff with fonts". If one program is better than another with "typography" (which might not be the best way to put it, but let's leave it at that), it means that the final result looks or "prints" better - that is, you have more control over the placement of the elements, text and graphics are properly aligned (and tenths of milimeters count here!), justification is both precise and flexible, etc. I have laid out scientific books in Word and did a decent job with it, but any professional could tell that the tool used could be better.

This is important, as my preferred solution i.e. Word mail merge tools, could work well enough for you, but the printed output could be unacceptable for other people involved (this might also be an issue with Access).

Why Word mail merge? Because it is dead simple. You could start with a simple template and a table of elements and build it up to direct Access access (sorry ) and macro-managed scenarios. I will not go into details, I think the help is accessible enough as it is - basically you set up one document with text placeholders and conditional fields and then feed it with another document, database, text file, etc. (By the way, Indesign also has a similar function - data merge http://tinyurl.com/c66x7m Maybe you should take a look at that first?)

Access seems to be very well suited for this as well. However, as I do not know it thoroughly, it would not be my first choice.

Finally, I would not dismiss xml - this is exactly the stuff was invented for. There are myriad solutions for going from database to xml and from xml to finely printed documents, you don't have to go fully DTD and xsl to do that any more (but you can).

[superboyac]

JohnFredC, thanks for explaining, it's really helping.  I need to hear your thoughts on this because most people who talk about access don't really use access.  you do, and that's why i need to hear more from you.  The people I've spoken to TRAIN access users, but don't necessarily use it themselves.  Others KNOW about how Access works, so they say what it can do, what I SHOULD be able to accomplish, but they don't really use it, so I can tell that some of my questions will remain unanswered.

FYI, our product is PRINT ONLY.  I think our information is simple and consistent enough where an access database can be used to hold everything pretty easily.  There are only 4 elements from what I can see, so it would just mean a few tables, some relationships, and that's about it.

I'm also pretty sure it wouldn't be too difficult to make a report that can output the format of the book as it is right now.  The part that's going to be hard (I think) is the fine-tuning and twiddling around the elements to make them look perfect.  I'm afraid there will be a few parts of the book where I'm going to want to manually move this diagram or question a little to the left or something, and I don't know how that kind of fine-tuning will work within the Access workflow.  Can you advise on that?

Jabberwock, I am also currently looking into some nifty xml solutions.  I like xml, and wouldn't mind using it at all, if it proved convenient.  I'm reading a little bit right now about xml and Indesign and what I can do with that.

And as for typography, I still don't really understand what that means exactly.  What does indesign offer me as far as typography that Access doesn't in a report?  Are the fonts, letters, paragraphs going to look better in indesign?  or does it refer to the ability to move things around and place them exactly where you want?  i really don't have a good handle on this.

[JohnFredC]

I'm afraid there will be a few parts of the book where I'm going to want to manually move this diagram or question a little to the left or something, and I don't know how that kind of fine-tuning will work within the Access workflow.  Can you advise on that?

I noticed in the online page-flipper that your diagrams and answers sometimes nearly overlap or perhaps might get in each other's way.  A couple of approaches would be:

1. Standardize the layout of each question/answer/diagram group so that overlap never happens.  This might cause an issue with larger diagrams, or may not be space efficient, but is definitely the easiest solution.

2. In Access, each band in a report can have VBA event code behind it.  Put some "OnFormat" or "OnPrint" VBA code behind the record (the report "band") on the report itself to move/resize the diagram under certain circumstances you define.  For instance, a question will be a tuple (ahem, record) of a query that joins fields from several different tables: Questions, QuestionTexts, QuestionAnswers, QuestionDiagrams, etc.  If you have created a boolean field in the QuestionDiagrams table called [MoveMeImTooBig], then you can have the question record inspect that field during the printing and act to reposition or resize the diagram (or the question text or the answer radio controls group or whatever) anywhere on the report surface based on the value stored there for the diagram (or even something that occurred in the previous record when it printed).  

What does indesign offer me as far as typography that Access doesn't in a report?  Are the fonts, letters, paragraphs going to look better in indesign?

 
If your concern extends to adjusting the kerning between letters or point by point adjustment of line spacing, then Access cannot do that.  A critical publisher of art books might find the Access output a bit coarse due to its inability to adjust kerning and line spacing increments.  I'm betting your customers won't notice any difference.

or does it refer to the ability to move things around and place them exactly where you want?

VBA code in an Access report band can do that based on criteria it finds in the tables you design.

Do this:  open Access, start an empty database, use the automatic wizards to generate one of the built-in table structures, a query based on that table, a form based on that table, and a report based on the query.  Then open each object into design mode, display the properties window, click on various controls/elements, and browse through the properties supported, including the events.  If that doesn't get your creative juices flowing...

[superboyac]

Thanks JohnFredC, I will start playing around with that.  Yes, i am not so concerned about kerning, character spacing, etc. right now, but I *may* be.  Is there really no way to adjust those properties in the text?  Even with all those options I see?

[wraith808]

I spent 15 years of my professional life making my entire living as an independent Access developer so please bear with me.

No offense intended to anyone here, but those who poo-poo Access probably haven't used it professionally.

Heh.  I've used it more than I'd wish professionally, and I can say that a good 40%+ of my time (and I'm being generous) was spent getting around limitations of it, whether with pure access development, or using access as a back end.  While it's true that some of my distaste is rooted in ODBC, Access can IMO be very easily misused.  That's not to say that with help it couldn't be a platform for the project presented here, but IME, outside of small-scale data entry projects, you quickly run up against it's limitations.  I've always thought this was the reason that MS has been marginalizing it as of late- because with misuse, it becomes a PR nightmare.

[JohnFredC]

Access with an Access backend (mdb) is wonderful for single person apps and good for small workgroups with a handful (say, less than 20) of simultaneous users.  Access as a front-end to SQL server or Oracle is a wonderful tool, plain and simple.

Some more "pithy" remarks: 

Friendly, powerful development tools and data do not an application make.  Good software is more about the UI metaphor, efficient data structure designs, data integrity, etc, etc. 

Access's brilliance is that it leverages any moderately intelligent person into data-driven application development. 

Access's downfall is that it leverages any moderately intelligent person into data-driven application development, even those who know nothing about application development.

DP departments most particularly hate it, because end-users tend not to be app-development savvy.  Give them Access, though, and they can make something that is functional for their department very quickly.  Then management wants DP to take over maintenance, extend the functionality, convert the app to a robust client/server environment.  Typically the original developer didn't know about relational forms, data integrity concepts, or efficient query structures, many-to-many tables, inside vs. outside joins, or what have you:  DP inevitably finds errors in implementation of major proportions.  Further, the kinds of functionality that Access enables with such ease (self-modifying reports, for instance, see my post above, or real-time self-populating pick-lists) are much more involved to create in the kinds of tools that DP uses and with the kinds of expertise that DP hires, adding to the budget needed to replicate the end-user designed functionality. 

So DP lobbies against end-user ownership of the data (and, by association, application development) via various (sometimes draconian) policies that restrict the use of end-user tools such as Access.  App development professionals who are qualified in other, DP-sanctioned tools are hired by DP, absorb the "bad" press, and repeat it without a second thought.  The result is that the line-of-business is stuck without the (sometime extremely simple) tools it desperately needs and in the fast-paced modern world, opportunities to enhance shareholder equity pass by.

Fortunately for developers like me, the more astute line-of-business management can see through the DP posturing toward the benefit of hiring a professional who can produce badly needed robust solutions very quickly (of course having DP miss its own deadlines over and over again doesn't hurt).

I've been in the business for 30 years on both sides of the fence: in-house corporate database development and delivery (very large unnamed bank) and subsequently as an outside (and successful) purveyor of those very apps built in Access.  We could debate the politics and best practices issues in another thread, but...

For the sorts of things that superboyac is inquiring about, Access is like a dream come true.

Whew, sorry for such lengthy opinionating! 

[wraith808]

Access with an Access backend (mdb) is wonderful for single person apps and good for small workgroups with a handful (say, less than 20) of simultaneous users.  Access as a front-end to SQL server or Oracle is a wonderful tool, plain and simple.

<snip />

For the sorts of things that superboyac is inquiring about, Access is like a dream come true.

Whew, sorry for such lengthy opinionating! 

We'll have to agree to disagree on that point.  I've been around the block a few times myself, and have dealt with a variety of situations from access being used as a back-end, to access being used as a front end for SQL Server, Oracle, and Sybase (you want to talk about a nightmare?  Access+Sybase.  Actually, anything and Sybase... but that's a different discussion).  And in my experience, Access and enterprise has been a headache.  Admittedly, that was a while ago.  But from a speed and configuration perspective, and getting around problems that enterprise solutions enabled me to solve more easily, Access never did me any favors.

[superboyac]

I sort of understand what is being said, and appreciate it.  But I do think my project is so simple compared to what is being discussed, that many of the points do not apply.  I am inclined to believe johnfred that access can do the job, because even before bringing it up here, I just couldn't see why it wouldn't be able to do it.
But I am going to try the two solutions here that I like:
1) trying to do this in Access
2) trying to do this using Indesign and xml

One of those has to work.  I may be wrong, but the very key issue is going to be the fine-tuning of the document.  Text styles, nudging images around, precision formatting (I'm unreasonably picky).  Plus, we've already done everything in Indesign by hand right now, so I might be leaning towards it.  However, this Access solution has me intrigued.  So much so, I even took a training class two weeks ago for Access.

[superboyac]

How about this workflow:
1) store all the data in Access (questions, solutions, diagrams, etc.)  maybe even create an input form to add new questions and such
2) export the content in the final structure as xml. (this is to avoid using Access reports as the final product, if it is not as flexible as I'd like)
3) import the xml into indesign, and use indesign to do the final tuning.

How's that?  I think that's a good solution in case access alone doesn't work.

[Stoic Joker]

As a network/systems administrator I'm definitely in wraith808's corner on this. I've seen many an Access based nightmare, and am thus a big fan of SQL.

But I will give you this one:

Access's brilliance is that it leverages any moderately intelligent person into data-driven application development. 

Access's downfall is that it leverages any moderately intelligent person into data-driven application development, even those who know nothing about application development.

Any time I use it the tied hands effect the WYSIWYG affords makes me want to tear my eyes out almost immediately. Only saving grace is that you can at least switch to a (semi) proper SQL Query prompt and get something done. But even that's a bit clumsy.

I've scratch written many of our internal systems, and while it did take more time to do it around a SQL db ... Once it was done, it was done. Scaling issues aren't a problem, the server just handles what is thrown at it. Which is good because I had a lot more time back when most of this stuff was done. So now as the data volume increases and usage picks up, I don't have to worry about it turning into a fire-drill because the db decided to max out and explode. Getting to the 50,000 record stability/performance turning point really isn't that hard to do these days.

I guess I've just never actually seen a good Access based application. *Shrug*