Author Topic: Capturing Scrolling Pages (Read 20,347 times)

SuperMan · « **on:** August 10, 2015, 07:14 PM »

I have an add-on for Firefox called qSnap. It's the only thing I've found that reliably renders perfect image snapshots of full web pages (even if they scroll on your screen) without actually saving the page in some html related format. It captures it instantly and lets you save it as an image and I've never seen a rendering error in any of the pages I have captured. I'm sure there must be other things that can do the same thing, but they are certainly not as easy to find as a lot of other software to perform basic tasks like that.

The screen capture programs I've used, including Screenshot Captor, use a method that scrolls the page and I'm guessing maybe stitches multiple screenshots together somehow as it scrolls? But I have found them to be highly finicky and unreliable and can't get them to work many times. Kudos to Screenshot Captor though that has many tweaks that allow you to adjust for different conditions to try to get the shot. It's the best attempt at that feature I've seen even though I still find I am unable to get reliable scrolls many times.

So my question must be obvious by now. Why can't Screenshot Captor use whatever method qSnap uses to convert a scrolled area to an image instead of this apparently different method that requires a finicky automatic scroll?

I'm guessing that the main reason is that Screenshot Captor has to have a generalized approach that works in a broad range of applications whereas qSnap only has to work in Firefox and therefore can use some other method specific to Firefox. But I'd like to know more about specifically why this can't be done that way in say Screenshot Captor. It's so difficult to find something that can simply and reliably take a snapshot of a full scrolling web page (or a scrolled page in other applications) and convert it to an image without all the tweaking and finger crossing! I don't mean to imply that it is a simple thing from a technical point of view, but from a users point of view it is so basic!

Any light you can shine on this would be welcome! It's always stumped me. ... but then I'm easy to stump!

PS: The Fireshot add-on for Firefox performs a capture similar to that of qSnap. Instant and perfect rendering every time.

mouser · « **Reply #1 on:** August 11, 2015, 01:22 AM »

Browser addons have a much easier job of capturing a scrolling image from a browser just by nature of the fact that they are integrated with the browser -- they can tell the browser what to do and have easy access to getting exactly the window contents, etc.

Have you tried the new Screenshot Captor "manual" scrolling capture method? It takes a few more clicks but once you get used to it the procedure is pretty straightforward and the results are predictable.

Regardless, i don't think there is any shame in using a browser addon/plugin to capture scrolling web pages, and use a screenshot tool in other cases.

SuperMan · « **Reply #2 on:** August 11, 2015, 03:02 AM »

As always thank you for your reply mouser! I have not yet tried that manual feature, but I will give it a whirl. Even though the automatic kind of scroll method doesn't seem to always work well I do think that SC's scrolling capture feature is a brilliant piece of work though. I think it gives you the best chance of getting the shot.

Usually, if I want to do a scrolling capture, it is in a browser. So yes, the add-ons are probably best for that for the reasons you gave.

To be fair, I run my browser in a sandbox. So there is an extra layer of mush that SC has to deal with. That may be part of the problem. The sandbox does cause problems with some things and maybe this is one of them.

The real challenge is to get scrolling captures in other applications besides browsers. It's so hard to get consistent results with anything. I guess that may be because each program works differently in one way or another and its hard to make a universal tool that can deal with every contingency.

mouser · « **Reply #3 on:** August 11, 2015, 03:08 AM »

Quote

The real challenge is to get scrolling captures in other applications besides browsers. It's so hard to get consistent results with anything. I guess that may be because each program works differently in one way or another and its hard to make a universal tool that can deal with every contingency.

that's why i finally added a "manual" method into SC's scrolling capture. once you get used to the way it works, it's hard to go wrong with it.

SuperMan · « **Reply #4 on:** August 11, 2015, 04:13 AM »

Know what? I need to do more work to understand this better. When I tried it this time, the automatic scrolling was working really well on Firefox (except for the fact that it is also capturing the whole firefox application window and not just the page being browsed). Manual scrolling also worked really well. I think that the problem I had may have had to do with the sandbox. I am getting an error message related to DDE data before the capture takes place. This may be the problem.

What I have not been able to figure out how to do is to capture just the browser web page window without the Firefox tabs and toolbars appearing at the top of each scrolled page. I'm thinking maybe this is the problem caused by the DDE failure? I never seem to get a chance to select just that window and not the entire application window.

But the exciting part is that Screenshot Captor is scrolling like a champ now!

Does any of this make any sense?

mouser · « **Reply #5 on:** August 11, 2015, 04:16 AM »

Quote

What I have not been able to figure out how to do is to capture just the browser web page window without the Firefox tabs and toolbars appearing at the top of each scrolled page.

the real trick to this is not to worry about it. after SC performs its scrolling and capturing, it has a button you push to detect where the real content is and crop out the rest.

SuperMan · « **Reply #6 on:** August 11, 2015, 12:11 PM »

OK! Now I understand how that part works! I didn't think of using it to eliminate the multiple tab and menu parts on each page of the scroll by adjusting the overlap.

Very very cool implementation!

It really is brilliant!

On my first test the right margin control didn't seem to crop anything. But when I tested it on another scrolling page it worked. I'll have to investigate that more. All the other margin adjustments worked great! And the Scroll Overlap controls are amazing!

One other small issue... How do you eliminate the red page separator lines from the capture?

But WOW! What a fantastic implementation of scrolling! Better than any other I have seen.

A possible enhancement.... Allow different zoom levels in this dialog other than Fit and 100%. Might make it easier to see what's going on while you are adjusting things.

Edit: Since you can't crop the overlaps of each capture individually, it's clearly critical that each capture has the same amount that needs to be cropped. Otherwise when you set the scroll overlaps so that one seam is perfect, another seam might not be right. On my test I did a manual capture using the page down key before each capture to try to make each capture uniform size. When I set the overlap based on the perfect setting for one seam, the other seams were off a little. Is there a way to avoid this?

Ath · « **Reply #7 on:** August 11, 2015, 04:55 PM »

Only the last page will be smaller then the others, when starting the capture from the top of the page, that's why there are separate settings for the last page, an also for the first page to cut off an unwanted header

mouser · « **Reply #8 on:** August 11, 2015, 07:06 PM »

Both SuperMan and Ath are correct regarding overlaps.

SC does assume that all but the last scroll will be of equal size. It is possible, though rare, to be using an app where the scrolling, even when done by "pages" is not exactly the same size on each page.. this can produce a problem where the overlaps may be off by a pixel when stitching. I can give some thought to some option that might try to detect that and nudge the overlaps a tiny bit if it can figure out where it happens..

SuperMan · « **Reply #9 on:** August 11, 2015, 11:52 PM »

mouser... truly I am so impressed by your grasp of topics like this! What an amazing talent!

Yes, that is indeed what happened in my case. It was not the top or bottom capture that was off after adjusting overlap, but one of the captures in between. When I adjusted one of the middle captures to have a perfect overlap, the overlap of one of the other middle captures was too high. It seems to me it was cutting off more than 1 pixel too much though, seemed like more than that.

I actually used this thread to do the test. At least one of the middle "slides" had maybe a several lines of text cut off while another "middle"
slide was completely perfect.

Thinking about it though, it seems possible that you may not be properly adjusting the overlap even for a slide that looks correct. What if there is white space for example around the overlap. How can you visually determine what the correct overlap is in that case to the pixel (if those units are pixels... not sure)?

So if you over reduce the overlap on one slide, that has say horizontal white space, and there is another slide that has data around the overlap it could cut off the data without your knowing. So A. maybe you should set the overlap on something you can be confident about if possible. and B. you should check all the overlaps even on the middle slides before finishing.

But actually I did try to do A although I'm not exactly sure what you can be confident about. I chose a slide with some split words right on the seam and tried to align the words so they were properly overlapped. This may or may not have been an accurate overlap.

I don't want to over complicate this. mouser I am sure you understand the issues. The goal, of course is how to get at least a visually "perfect" image with the minimum amount of work. With the amazing tools in Screenshot Captor it's already pretty close!

If it would be helpful at all I can try to reproduce an image with this issue and attach.

Also, if you get a chance, do you know why the red margin/overlap lines appear in my final image and how I can prevent that?

Added:
I tried to capture and image this thread again applying some understanding I gained in this discussion. Margins don't seem to be any issue in this test, it's the overlaps that are difficult. The automatic overlap didn't work so I adjusted it manually. I tried to be very careful to align a middle section that seemed clear. At first it seemed to be pretty good when I checked the other middle slides... but then, after adjusting the End Overlaps, I looked at the middle slides again and something seemed way off. I don't know what I did wrong as I assume the End Overlaps shouldn't affect the middle slides?

Anyway, I need to do my tests more scientifically when I have more time so I can be more succinct in what I write and not waste everyone's time. I apologize for the long posts without much substance. When I have more time I will try to report what is going on in a way that can be reproduced... or else figure out what the heck I did wrong!

This is an awesome feature. Hope I can add something to make it even better! "I'll be back!"

IainB · « **Reply #10 on:** August 12, 2015, 05:55 AM »

If the otherwise excellent SC is not playing nicely with a scrolling capture of a web page, then this might be of use/help: whilst I'm not usually interested in capturing images of scrolling windows (web pages), it does seem that the OneNote Clipper bookmarklet "button" - as used with the "FREE" MS OneNote - still works very well in doing a quick and flawless image capture:

Quote from: IainB on March 19, 2014, 09:35 AM

Test of the OneNote Clipper bookmarklet "button" that you drag to Favorites:
My test results indicate that this takes an image of an entire web page (a scrolled-window image, much like Screenshot Captor, but without all the palaver associated with the latter), but:
(a) The image is of the entire web page, regardless of whether you have only selected/highlighted a part of the page.
(b) It doesn't seem to work in Firefox v28ß (could be my Firefox settings, I suppose).
(c) It works perfectly in IE11.

The way I work, my objective is usually to save selected parts, or all, of a web page in HTML format and often with attached/nested pages/files.
Thus I rarely take such images/screenshots, and the OneNote Clipper is not of much use to me.
However, when I want to capture an image of an entire scrolled web page, in future I shall consider using OneNote Clipper rather than SC (if I remember).
So, I shall continue to use Firefox with the Scrapbook extension for capturing part/all of a web page in HTML format (having come across nothing better with a non-proprietary format, or greater reliability so far).
______________________________________

Using the OneNote Clipper to capture the webpage, for example, here's this thread page in an image: (click to enlarge)

[attachthumb=#]

SuperMan · « **Reply #11 on:** August 12, 2015, 06:33 AM »

Thank you for the tip. I'll check it out!

PS: Major American Monty Python fan here.

IainB · « **Reply #12 on:** August 12, 2015, 09:46 AM »

Quote from: SuperMan on August 12, 2015, 06:33 AM

...PS: Major American Monty Python fan here.
_____________________________

I suspect there's a lot of us about.

SuperMan · « **Reply #13 on:** August 13, 2015, 03:13 AM »

Can someone tell me how to avoid having the red margin and overlap lines appear in my final images?

I'm not sure if that is something I am doing wrong, a bug, or an environment issue on my computer.

Thanks!

tomos · « **Reply #14 on:** August 13, 2015, 08:19 AM »

Quote from: SuperMan on August 13, 2015, 03:13 AM

Can someone tell me how to avoid having the red margin and overlap lines appear in my final images?

I'm not sure if that is something I am doing wrong, a bug, or an environment issue on my computer.

any chance of attaching a screenshot?
- and describing process used.
Are the overlap lines red too? (Or are they just overlaps?)

I just tried a partial manual scrolling capture of this page (which I also cropped top and bottom after) -
see attached. I dont have either problem here. [edit] I see too much overlap was stripped out in screenshot already in the first post - I had missed that... [/edit]

SC 4.12.0
Win 7 x64

SuperMan · « **Reply #15 on:** August 13, 2015, 03:08 PM »

Thank you for the help!

Wow! This is kind of funny!

1. So I did a quick scroll of this thread to make a screen shot to post to demonstrate the "red lines." I used automatic scrolling (SC tried a few methods and selected one) instead of manual. Then I used the automatic margins and automatic overlap. I have not inspected the result thoroughly, but it appears that it came out PERFECTLY ... both margins AND overlap. In addition, for the first time ever for me, the scroll only contained the browser window, not the Firefox page tabs or command menu etc. above the browser window. That is what I wanted to achieve but I don't have any idea why it did that this time and not the other times. Weird. I'm going to have to experiment with that to see if I can figure out what's going on.

2. The red lines are not related to SC! As I was doing this I finally realized what they are. I run Firefox in a sandbox. The sandbox puts lines around the Firefox window. But the way I have it set up they are barely visible and I am so used to them I forgot they are even there.

Now... when you use the SC editing window for setting margins and overlaps, you can see the red lines around each slide. Just coincidentally, they appear in the perfect place around each captured slide because they surround the captured window. So all along I assumed these lines were part of SC... there to help you adjust things so you could easily see the edges of each individual slide! Actually, I thought that was a very useful feature! The lines are red and partially transparent.... perfect for seeing where the edges are with precision as you adjust them!

So I assumed those lines were part of SC. I didn't realize they were actually part of the capture!

This has to be one of the oddest things I have ever seen when testing something like this.

But honestly, I think the semi-transparent red lines around the slides in the stitching dialog are a very useful feature... maybe something you might want to consider! ... of course they shouldn't appear on the output.

Bottom line.... the red lines are my mistake.

But maybe not a bad idea!

I attached the slide of the output image anyway to show you what I mean. As you can see, the red lines appear at the end of each break. It was even useful having them in the output because it allowed me to inspect the overlaps quickly as I could see where the page breaks were. But of course, except for testing purposes, I wouldn't want them in the output... only in the editing dialog. On further inspection it appears that the "internal" overlaps look perfect! ... and those are the ones I had problems with. The last page is off, but that's not an issue. But how everything worked automatically this time and not the other times, and how this time it only captured the browser window and not the application window is still a mystery to me.

But pretty cool!

Follow up:

Try as I may I have not been able to reproduce the scrolling capture of just the browser window that I did once and now can't repeat. No matter what I do, every scrolling capture I do now includes the entire Firefox application and not just the Firefox browser window. I even tried it out of the sandbox but got the same results.

Here is what I am doing:

1. Ctrl+Shift+PrtScr ... The Red SC dialog appears in the upper left of Firefox.

2. From the "gear" icon on the left of that dialog I select "Capture Selection Now"

Except for the one time I did it, when only the browser window of the Firefox application window was captured, the entire Firefox application window is captured every time. So each separate frame includes not only the browser window but the entire Firefox application screenshot. I don't know what made the difference.

Is there any way to select which window is captured on the scroll to intentionally select only the window that scrolls?

tomos · « **Reply #16 on:** August 13, 2015, 07:17 PM »

Quote from: SuperMan on August 13, 2015, 03:08 PM

In addition, for the first time ever for me, the scroll only contained the browser window, not the Firefox page tabs or command menu etc. above the browser window. That is what I wanted to achieve but I don't have any idea why it did that this time and not the other times. Weird. I'm going to have to experiment with that to see if I can figure out what's going on.

in my manual scroll above, they didnt show - in the past I have gotten them at times, but never really tried to figure it out.
(FWIW, if you have tabs showing at the bottom of the window, they always show in the shot IIRC)

Glad you figured out the red lines

SuperMan · « **Reply #17 on:** August 14, 2015, 12:27 AM »

OK! In deference to my ?British? (assumption) friend IainB, I think I sussed this one out!

In case I have not confused things enough with my "red lines" conundrum, wait until I get into this before you forgive me.

The red box that appears on the page when you press Ctrl+Alt+PrScr is a control called "Red Box Capture." It's brilliant!

It allows you to select individual windows objects. It has two modes.

From the help file:

Quote

You can switch Red Box Capture operation between a "region" mode, where the outline can be moved and resized by your manually, and an "object" mode, where the outline follows your cursor around and selects the windows object (like menu, toolbar,window,panel) under the cursor at any time. Use Shift+Left Mouse Click to switch modes, or else just double click on the red+white text.

Now here is the stupendous cosmic befuddlement!

It outlines the object to be captured *with a red line!!!!* Yep. It's a coincidence upon a coincidence upon a coincidence! It's Deja Vu all over again!

So there are yet other red lines here in my case! No wonder I was confused (through no fault of SC!).

But now I think I got it.

Apparently object mode is not supported in Firefox... least it didn't work for me. So in order to select scrolls that only include the browser window I had to switch to region mode and manually move the top of the red outline down to the top of the browser window. Guess what? It works like a charm! You can then do the scroll and the top will start there on every frame. How sweet!

I can't explain how it did that for me automatically that one time, but I guess it's just one of the unexplained mysteries of the universe (better known as sniglets).

But the key is that it can be done!

I had to use a manual scroll though. SC couldn't find an automatic method. Still, in the end I got it. NICE!

Question: Is there a text scraping tool? It is referenced in help, but I can't find any info about it.

mouser · « **Reply #18 on:** August 14, 2015, 04:09 AM »

It sounds like you've found a bug with the red-lines being used to outline the area for capture -- I'll try to fix.

Question -- did those red outlines only happen when the window you were capturing was full screen?

Ath · « **Reply #19 on:** August 14, 2015, 08:42 AM »

Quote from: mouser on August 14, 2015, 04:09 AM

It sounds like you've found a bug with the red-lines being used to outline the area for capture -- I'll try to fix.

Question -- did those red outlines only happen when the window you were capturing was full screen?

I think he said it was a feature of the Sandbox software, not caused by ScreenShotCapture:

Quote from: SuperMan on August 13, 2015, 03:08 PM

2. The red lines are not related to SC! As I was doing this I finally realized what they are. I run Firefox in a sandbox. The sandbox puts lines around the Firefox window. But the way I have it set up they are barely visible and I am so used to them I forgot they are even there.

mouser · « **Reply #20 on:** August 14, 2015, 08:45 AM »

ahhhhhh i see

always nice to hear when a problem is not my fault. doesn't happen often

SuperMan · « **Reply #21 on:** August 14, 2015, 07:27 PM »

Zounds!

Totally not your fault or an SC bug!

The coincidences are just too much!

The Red Box Capture tool puts a red line around the captured object which in this case was the whole Firefox application window.

The sandbox I am using also puts a red line around the whole Firefox window.

The stubborn red line I was seeing around the windows stitched together in my images was actually the red line from the sandbox program which Screenshot Captor *correctly* included as part of the capture, NOT the SC red line.

Talk about a funny coincidence!

But....

I also noted above that the red lines that appeared in the window where you stitch the scrolled slides together (which were actually the lines from the sandbox) were really useful in that window because they showed you precisely were the edges of each slide was. That makes it easier to properly space them. So I suggested that maybe putting some semitransparent red lines around the slides in the stitching window might be a nice enhancement!

Sorry I caused so much confusion about all this! $:-\$

mouser · « **Reply #22 on:** August 15, 2015, 04:37 AM »

Quote

I also noted above that the red lines that appeared in the window where you stitch the scrolled slides together (which were actually the lines from the sandbox) were really useful in that window because they showed you precisely were the edges of each slide was. That makes it easier to properly space them. So I suggested that maybe putting some semitransparent red lines around the slides in the stitching window might be a nice enhancement!

you know that is an interesting idea.. like a button to toggle semi-transparent red lines showing the stitching locations in the preview, so you could very quickly identify the overlap areas for inspection. neat idea! putting it on my todo list.

SuperMan · « **Reply #23 on:** August 15, 2015, 04:41 AM »

Perfect! Thanks mouser!

They say some of the greatest inventions of mankind were initially the result of a mistake.

(that must mean I've had a lot of great inventions!

)

The one that I know of was when the 3M company was working in their lab trying to create a formula for a strong glue. It failed because it was so weak it could hardly hold anything together. So they were ready to scrap it and start over when one of their crew had an idea. Hence the birth of sticky notes!