Looking to do high quality video conversion

[Carol Haynes]

Yeah, working with video takes forever! I would imagine for the most part any DVD9 to DVD5 compression will take roughly the same amount of time, but I'd be willing to bet that Nero is one of the fastest (it's probably somewhere in the middle for quality though).

-Hirudin (June 07, 2007, 06:47 PM)

I haven't tried NeroVision since version 7 came out - but they will have had to improve it massively since version 6 to make Nero a contender for quality. Version 6 recoding was utterly crap (IMHO) and suffered horribly from video/sound sync issues.

[Darwin]

Well... didn't get to that last night, but ripped a dvd while I dropped my boys off at school. As I left DVDFab indicated it was going to take over two hours but when I returned about 1 hour later it was done. Generates a 1 GB avi file from a regular (4.7 GB) dvd. Not bad. As Hirudin noted, I've got two weeks to rip five or six more and all is well.

[Hirudin]

StaxRip thoughts after trying it out extensively:
Well, it has a lot going for it! Most of which I mentioned in my previous StaxRip post. I have since discovered a couple more very cool features... 
It has this "Event Commands" tool that lets you "do stuff" after other "stuff". A simple example that I created for myself is a little "Remember to load the chapter file" message box.
It will automatically load 1 or 2 audio tracks (or 0 tracks I suppose). It'll also load up subtitle files (.srt, or maybe something else) based on the languages you choose. (So if you have English subtitles for the movie dialog and English subtitles for the commentary it'll load both!) A real time saver.
There is a pretty cool AviSynth editor thing. If you have an AviSynth function you'd like to add (say "RemoveGrain") you can add it to your profile template very easily.

Unfortunately it's got a few flaws too...
Although the "Event Commands" feature is cool, the execution is poor. I had lots of cool ideas for this, but was not able to do much more than the most simple... For instance, I wanted to set up a command to set the output filename based on the input directory name instead of the input file name. 22 files all named "VTS_01_1.mkv" isn't very useful...
Oops. I should have looked at the VERY FIRST tab under View -> Options
I very much think the program is geared toward creating MKV files, but although it opens the audio and subtitle files automatically it leaves loading a "chapters" file up to the user... Doing 1 file is fine, but if you're doing a couple seasons of a TV show it gets a little annoying. Also, it's too easy to forget to add the file as it's inside another submenu thing (container configuration). --- I'm going to try to write an AHK script to automate this...

It's funny, I started this post to try to tell people to avoid StaxRip... But I've convinced myself that it's worth at least one more look...

[Hirudin]

Welp, I've been plunking away at making some small, high quality videos. I've done a lot of trial and error testing and here's what I've come up with... I don't know how useful this info will be to any of y'all, but I feel like it's selfish not to share...

StaxRip is indeed pretty great. Sure, there's a couple things I'd prefer to have automated, but that's what AutoHotkey is for I guess.

AviSynth is definitely the way to go for video filters. I'm sure there's stuff out there that rivals what you can get for AviSynth, but I'd be very surprised if you could do much better for free... Here's a couple filters that I've found useful...
FFT3DGPU (noise removal) - this is awesome! It's a remake or something of FFT3DFilter, but it uses your video card's GPU to do the processing (which speeds it up considerably). I'm getting 4 times the performance from my GPU (a Nvidia 7800GTX) than my CPU (a AMD X2 4800+) with better quality settings!
Command I am currently using: FFT3DGPU(bt=4, plane=4, bw=32, bh=32, precision=2, mode=2)
FieldDeinterlace (deinterlacer) - There's probably better deinterlacers out there if you're willing to check your files manually. I didn't feel like going through the hassle. This seemed to work pretty well, and it didn't demand setting lower/upper field first manually for every video.
RemoveGrain (noise reduction) - By itself "RemoveGrain(mode=4)" helped lower compression size considerably without changing the picture very much at all. This one doesn't seem to do a whole lot, but running it after the deinterlacer seemed to change very little, but be helpful overall. As this plugin can reduce the appearance of interlacing it's better to run an actual deinterlacer before this.
Lanczos4Resize (resizing) - I did some comparison testing between the following resizers (most standard with AviSynth I think): Bilinear, Bicubic, Gauss(67), Lanczos, and Spline36. I didn't spend too much time tweaking (most of them don't have any tweakable settings), but Lanzcos4 seemed to produce the best quality for my purposes: reducing 720x480 4:3 DVD content to 640x480.

My Full AVISynth Script

MPEG2Source("<path>\VTS_01_1.d2v")
Telecide(Guide=1, post=1, gthresh=1)
Decimate()
FFT3DGPU(bt=4, plane=4, bw=32, bh=32, precision=2, mode=2)
FieldDeinterlace()
Removegrain (mode=2)
Lanczos4Resize(640,480)

ChapterXtractor works real well to get the chapter times from your IFOs. If you're making an MKV with StaxRip an OGG formatted chapter.txt file works just fine. With the path to the IFO you can run: ChapterXtractor.exe <IFO file path> <output .txt file path> -p7. (I think the OGG format is preset 7 (hence the "-p7"), check the ChapterXtractor.ini file...

SubRip is the only way I could find to convert DVD subtitles to .srt text files. It uses OCR (optical character recognition) like scanner software to "read" the subtitles. It needs to learn every character you run through it, so it can take a while until it's learned pretty much every letter (capital and lowercase), number, and symbol.
Here's an AHK script to run SubRip and ChapterXtractor when the path to an IFO is copied to the clipboard. This does a lot of mouse clicks and goes  from opening SubRip to saving the file to running ChapterXtractor. It's possible you'll have to edit every set of "Click" coordinates, so expect to do some editing...
sub+chapx.ahk (3.29 kB - downloaded 315 times.)

I've also got a big script to load files into StaxRip very quickly (almost entirely automated mouse clicks) but it requires special settings in StaxRip and also it's very specific to what I'm doing (for instance it clicks the "down arrow" a number of times because of how many templates I happen to have saved). If anyone out there wants it I can upload it on request...

Also, sorry for not making the names of the programs and stuff links... I'll try to get to it soon... For now Google will have to suffice.

[Hirudin]

And the final results? VERY impressive if I say so myself.

Here's the original DVD. It's being resized to 640x480 by The KMPlayer, but that's it. I honestly don't think the quality is affected in the slightest.
Looking to do high quality video conversion

Here's an Xvid-AVI that I made with DVD Rebuilder Pro. It looks just fine. I didn't spend too much time messing with the settings, but there weren't many either.
Looking to do high quality video conversion

And the crowning jewel: The x264 MKV! No, I didn't mix-up the pictures, the final encoded file looks BETTER than the original DVD (in my opinion at least).
Looking to do high quality video conversion

You might be wondering, what about the file sizes...
DVD: 876 MB
AVI: 299 MB (This only has 1 audio track and no subtitles) (Encode Time: ? ~15 minutes or so? I can't remember. ?)
MKV: 206 MB! Sweet! (Both audio tracks and 2 sets of subtitles) (Encode Time: 124 minutes, wow)

I'm quite proud of the results! Was it worth my time? That's debatable, but it does look good.
[edit]Oploaded screen caps that include some subtitles. God DVD subtitles are ugly![/edit]

[wr975]

Thanks for sharing your results, but it seems you quite abandoned the "easy way" of video encoding. You've listed a lot of manual steps. 

Considering the encoding time I'd personally opt for the XviD version. The quality is hardly worse but *much* faster encoded. After all, when all is done and ready you watch the encoding on your TV and don't sit 40cm away from your monitor looking for artefacts. I think with 200 MB it wouldn't look much different.

About subtitles... be sure to use the OCR correcting feature of SubRip. It'll fix many common OCR errors. I also use Subtitle Workshop (#1) for additional OCR correcting + time sync. For displaying subtitles in movie players I use the DirectVobSub filter (#2), as it offers a lot of options how to display text subtitles (font, border, shadow, color, background, size, transparency, spacing, position...)

#1: http://www.urusoft.n...hp?cat=sw&lang=1
#2: http://www.free-code...oad/DirectVobSub.htm

[Hirudin]

Thanks for sharing your results, but it seems you quite abandoned the "easy way" of video encoding. You've listed a lot of manual steps. 
...

-wr975 (June 14, 2007, 04:02 AM)

Heh, that I did! AutoHotkey does a lot of the work for me now though... But the setup was long and arduous... 

...
About subtitles... be sure to use the OCR correcting feature of SubRip. It'll fix many common OCR errors. I also use Subtitle Workshop (#1) for additional OCR correcting + time sync.
...

-wr975 (June 14, 2007, 04:02 AM)

Yes! Thanks for mentioning that. I actually went through and opened each in Open Office to spell check them, but I still had to do a lot of manual corrections. I'll definitely check out Subtitle Workshop!


The latest development for me is that my computer is having some kind of problem with the filters (I suspect FFT3DGPU). I'm getting odd red and blue leading/ghosting with some movement.
Looking to do high quality video conversion

I think to save time I should probably look into exporting the filtered video to some kind of uncompressed format or something...

[Hirudin]

I did a quick try of outputting the frames to individual files after the filters, planning to then use the frames again to encode the MKVs and stuff. 2 problems came up:
1. The frames will take a lot of space. 640x480 frames were taking 450k each. 24 FPS * 60 seconds * 450k = ~540MB/minute
2. So far, it looks like doing this will actually increase the encoding time.

I can live with #1, but #2 just sucks! My theory went a lil' something like this: Outputting the frames will take a long time (my computer seems capable of ~15 FPS) but I'll only have to do it once. After the initial frame output every subsequent encode would go relatively quick. I was hoping for at least 30 FPS, but anything over 23 FPS would yield a net speed increase for a 3-pass encode as I've been doing.

[Hirudin]

Another update...

First, I took "the easy way" out of the subject for this thread. I'm not thrilled about it, but "easy" certainly isn't part of the equation for me any longer... 

Also, I've found a lossless video compression codec called Lagarith. It's free, it's apparently fast (though I haven't tried others for myself), and it supports multi-threading. Encoding speed for my first, test video was about 17 FPS for Lagarith and so far it's about 37 FPS for x264 (first pass).

Math

Lets see, the video I'm working on has almost 34,000 frames (~24 minutes).
34000/17 = 2000
34000/37 = 920 or so (3 passes at this speed)
2000 + (920*3) = 4760 seconds of encoding.
Using the old method I was getting about 15 (<14 really) FPS.
(34000/15)*3 = 6800 seconds

If my math is correct, this new method = 30% faster... AND the source video will be exactly the same for every pass, increasing the efficiency, hopefully producing smaller files. The speed increase will be amplified since I'm actually encoding 2 sets of files (1 for computer use, 1 for PMP use) for a total of 5 passes.

The lossless video is about 4.4GB by the way.

- second pass is going at 43 FPS, sweet!
- oh no, the 3rd one is back down to 15...
This puts the total speed increase for 3-pass x264 640x480 video at ~12.5%... not that much... I wonder how 2-pass 320x240 xvid will fair...

[Hirudin]

Sorry for doing a monologue here... I'm getting burned out on video, but maybe this'll help save someone some time, or get someone some higher quality video encodes...

Sometimes I skip steps when I do stuff like this, a step I skipped here is "what the heck is all this YUV/YV12 gobbildygook everyones's talking about? Well, This Website explains it all... or at least it explains enough that I feel like I know it all...

A particularly interesting quote...

VirtualDub (and variants) run in RGB mode when you use Normal Recompress or Full Processing Mode (in the Video dropdown menu). All of VirtualDub's internal functions and filters run in RGB colorspace only. However, Fast Recompress doesn't decode the video to RGB, and instead just passes whatever your source is into the compressor you've selected - thus if your source is a YUV type then it passes the video data as YUV into the video compressor.

The rest of the page is an interesting read, check it out.

So a person like me who sees "Full processing mode" shouldn't automatically assume they're getting something better...
Looking to do high quality video conversion

I wonder what effect this'll have on my encodes, if any.

[mikiem]

Really, REALLY FWIW...

I've been tinkering with video on PCs for better than a decade now... That doesn't allow me to make any claims to be an expert or anything -- & I don't! 

What I do want to add is a bit of experience: It's VERY possible to screw up your windows installation's video handling if/when you install any do-it-all tool, usually because they often register a bunch of filters, very often the same ones (though different versions). The more video tools you have, the more risk involved. At any rate, I'm not saying don't use them, but rather use a bit of caution and do things like back up the drive with windows. Problems might be rare, but when they occur (& you can read about them in most video forums), without a backup it often turns out the cure is a complete windows re-install. 

Otherwise there are a Bunch of front ends listed at the video sites like videohelp.com that will do the work for you, most using the same freeware libraries and filters, many without requiring any install & so involve less if any risk. I wouldn't want to attempt a recommendation 1) because I've got a fair number of tools and I've experienced conflicts making it impossible to use *many* -- 2) over the years I've generally gotten so used to it I actually prefer the step by step routine. 

Finally, the closest I've seen to a (hopefully) good solution to go directly from whatever to mp4 is actually bundled with Roxio's EMC9 which includes copy to DivX (& several other formats) from whatever (including DVD). Folks on their mailing lists (think I got there because of the EMC9 trial) have been almost spammed with offers from Roxio for their Deluxe suite at $45 US, & a $30 (very liberal) competitive upgrade rebate is available. Do the math, check out the package, which includes a Dazzle DVC 90... maybe it'll fill the bill.

[Hirudin]

I thought I might have screwed up my codecs and stuff at one point through this, it was after I installed the lagarith codec, nothing wanted to play. I forget what I did to fix it though, I think I just installed Xvid again or something... Or maybe I just restarted my comp? The problem didn't last very long so it's hard to remember the fix.

I definitely prefer the quality I get from the step by step routine, but to me the step by step by step by sept yb stpe vy esiop routine gets old.

Here's a couple more things I've learned...
There's an AVISynth filter called "SSIM" which uses an advanced seudo-subjective algorithm to compare 2 videos. It generates a score for every frame; "1" means the frames are identical, "0" means they're completely different. It takes all these scores and spits them out as a CSV (spreadsheet) file which can be used to create graphs. It's pretty cool! You can even compare 2 videos to a source video at the same time using other AVISynth functions...

Like Dis

A=AVISource("F:\!TV - Seperated\Seinfeld\05\0519 + 0520 The Raincoats\0519 + 0520 The Raincoats.avi").Trim(0,61426)
B=AVCSource("F:\!TV - Seperated\Seinfeld\05\z-Compare\0519 + 0520 The Raincoats - 19.dga")
C=AVCSource("F:\!TV - Seperated\Seinfeld\05\z-Compare\0519 + 0520 The Raincoats - 3 Pass.dga")
StackVertical(ssim(A,B,"F:\!TV - Seperated\Seinfeld\05\z-Compare\0519 + 0520 The Raincoats - 19.csv","F:\!TV - Seperated\Seinfeld\05\z-Compare\0519 + 0520 The Raincoats - 19.txt",lumimask=0),ssim(A,C,"F:\!TV - Seperated\Seinfeld\05\z-Compare\0519 + 0520 The Raincoats - 3 pass.csv","F:\!TV - Seperated\Seinfeld\05\z-Compare\0519 + 0520 The Raincoats - 3 pass.txt",lumimask=0))

Looking to do high quality video conversion

x264 single pass "constant quality" produces very good quality video. I used SSIM to compare different quality levels to an existing 3 pass encode (set to "75%" in StaxRip). Quality:19 produced almost identical scores (almost always very slightly higher) and almost identical size (slightly smaller (353MB instead of 356MB)). To save a little space (with virtually 0 quality loss) I'm going to use Quality:20 (this particular video dropped to 298MB).

For some odd reason using the same source file, the same settings, with the same version of x264 (661) (the same everything) I'm getting slightly different encodes. It's definitely not enough to notice, but Beyond Compare and SSIM can tell. I don't care very much, but it does bug me a little... I didn't think this was suppose to be possible with com-poo-tars. It's as if x264 has a random number generator in it somewhere...

There's a tool and plugin for parsing AVC video into AVISynth. It's quite handy... It's made by the same dude that wrote DGIndex (formally DVD2AVI). It's called DGAVCDec and works exactly the same way.

[Hirudin]

Well, I'm giving up on StaxRip. It's sad, I thought it was the shit, but I was wrong...
The final flaw was that I cannot easily set the aspect ratio of the container. Since my main goal is "high quality" I've decided that since I'm using DVD video (which is always 720x480) as the source I might as well encode the final video at 720x480 and set the aspect ratio of the container (mkv in this case) to whatever it should be.
It is possible to change the aspect ratio in StaxRip, but you have to edit the command line sent to the muxer. I'm fine with that, but it doesn't seem possible to set it for a template (so it would have to be edited while setting up every encode).

So, the hunt continues...

[edit]Actually, going to postpone the hunting and go for creating. I wonder how much easier this can be using the command line interface of these programs and AutoHotkey...[/edit]

[mikiem]

In case it helps...

While the orig aspect is .9 something in a 720 x 480 frame, you might not want that depending on your target viewing hardware. An orig 720 x 480 (D1) mpg2 will display correctly on a std TV, & DVD Player software will alter the aspect of the viewing window to compensate. However the same frame in another format will often not show correctly on a PC or a compatible set top player -- perhaps rather than trying to edit the aspect ratios of the newly encoded file it would be better to [optionally crop &] resize the frame correctly? AFAIK the main purpose of setting PAR in a mp4-type file is for the correct display of various wide screen ratios without any letterboxing (as you often get with 16:9 DVDs). Encoding black letterboxing is not good with mp4.

If you go the RYO (Roll Your Own) route you might find a combo of DGIndex, Avisynth, & possible VirtualDub handy. Avisynth is all about template files; DGIndex will parse the mpg2, create a template Avisynth .avs file, demux & convert streams including AC3 to .wav (for re-compression); if needed V/Dub can host the encoder. All are open source, have cli, and are widely used in this sort of batch program.

OTOH I think one of the hard parts to automate is if you've got a letterboxed wide screen frame, detecting the amount of letterboxing in order to crop. If you're working with broadcast recordings, you might want to look at including comskip. If playback on a set-top DVD Player is not an issue, perhaps Nero Recode would work well for you? The downside is that it requires the Nero player for post processing to look good. The upside is it's extremely fast (20 -25 minutes for 1 hr 45 min  video), and you might find the OEM CD that includes Recode online for ~$5 US. It'll handle or let you specify the crop, resize etc. & can include subs.

[Hirudin]

Thanks for the advice again! It's clear you know what you're talking about on the subject!

...
AFAIK the main purpose of setting PAR in a mp4-type file is for the correct display of various wide screen ratios without any letterboxing (as you often get with 16:9 DVDs). Encoding black letterboxing is not good with mp4.
...

-mikiem (June 24, 2007, 03:20 PM)

First, to fill in anyone who doesn't know: PAR is Pixel Aspect Ratio. If the pixels of a dvd video were square the image would look squished/stretched (US-DVDs have a 3:2 aspect ratio), so DVDs have a "PAR". The pixels of a 4:3 DVD have a PAR of 8:9, pixels of a 16:9 (non-letterboxed) DVD have a PAR of 32:27.

The main reason I'm going to keep that PAR is to maintain quality (no scaling until playback). But avoiding letterboxing is also a very good reason for PAR.

I haven't had to crop anything yet, but I think I'll probably crop the letterboxes out completely and resize it slightly so that each dimension is a multiple of 16.

Regarding Nero, I've waved bye bye that that POS a long time ago, and I refuse to believe any "one stop solution" can offer anywhere near the quality:size ratio I'll get by using AviSynth and a little elbow grease.