Microsoft "Hyperlapse" tech is finally available to play with

[JavaJones]

I am shocked - shocked I say! - to see that there appear to be no previous threads on the whole "hyperlapse" thing (according to the Search at least). This is a variation of timelapse where the camera actually moves *large* distances (as opposed to the small dolly shots of most timelapse) while maintaining (relatively) smooth motion, creating an amazing combination of realistic and surreal imagery and motion. It really just has to be seen to be appreciated. This is an excellent introduction to what can be achieved:
https://vimeo.com/102841196

Like many things this is a technique that has been around for quite some time (earliest example I've seen was shot on film in 1995!) and was pioneered by some innovative photographer/videographers, painstakingly investing tons of time and effort into getting good results. And as with most great artistic innovations it is now starting to become more achievable for the average person who *doesn't* have days or weeks on their hands to plan, shoot, and edit such complex projects.

We first saw tools that anyone could use to create Hyperlapses from Google Maps street view data, which produced some cool results in itself. But the image quality and consistency were of course limited and the subject matter even more so. And whatever you did, it just wasn't *personal*, it wasn't *your* video.

Enter Microsoft Hyperlapse Pro!

Microsoft began doing research in this area a few years back and showed some tremendously promising results processing average GoPro-style mounted action camera videos into highly watchable compressed versions of the journey the camera captured. Rather than watching an hour long rock climbing expedition on a head-mounted camera, you can watch it in 60 seconds, with a fluid impression of the environment much as in the hyperlapses shown in the video above. This was a fairly revolutionary idea and the results of Microsoft's research really have to be seen to be properly appreciated:
https://youtu.be/SOpwHaQnRSY?t=1m4s
Unfortunately, while MS's research was promising, there was no software to go with it...

Well, I've had a web change detector watching their page for over a year now, waiting for the actual availability of software that implements their seemingly cool tech, and at long last it's available! GHacks has a good write-up:

Microsoft Hyperlapse Pro can be downloaded from Microsoft's Research website. It is compatible with all recent versions of Windows and only available as a 64-bit version.

The installation is straightforward and the installer itself is clean and does not include any surprises.

The hyperlapse video creation process itself is divided into four parts. First thing you do is create a new project and import a supported video format. Hyperlapse Pro supports mp4, mov and wmv video files only.

Unfortunately it does come with a watermark currently, which is a real shame, but it's still cool to be able to play with the fruits of their research. Instagram came out with a similar processing technology in an iOS-only app about 8 months ago, so this kind of thing has been available for a while already. However Instagram's approach is not as thorough or capable as Microsoft's seems to be, and of course it's iOS-only. Microsoft has the PC application as well as an option for both Windows Phone and Android owners to play with.

So does anyone have any videos they can try this one? Show us your results!

from Ghacks.net

[4wd]

http://research.micr...s/ivm/technology.htm

There's also an updated version of the Image Composition Editor available.

Have to try it on a 25 minute 4x4 video I made, I sped it up ~6x to get it down to the music length .

[JavaJones]

Yeah, ICE (Image Composition Editor) is also really cool tech, and is actually usable (no watermark, I think). It works better than many other apps I've tried for stitching, including Photoshop (panorama stitch mode).

- Oshyan

[4wd]

Guess I'm going to have to wait until they fix it, currently I can't get it to load anything.  It grabs >30% CPU, becomes unresponsive, and just keeps chewing memory until the system complains, then I have to kill it with Task Manager.

EDIT: And here's the answer:

Here are two things to check:

    Have you ever run Windows Media Player (WMP) on your system before?  Hyperlapse Pro uses components from WMP, and you need to run WMP at least once and accept the license agreement.
    Are you running over Remote Desktop Connection?  If so, check out the "Known Issues" sticky at the top of the forum for instructions on how to make video preview work over remote desktop.

Never used it, never had it installed - honestly, you think they could mention it in the requirements.

[TaoPhoenix]

Or just borrow the components of WMP and unite it with their new project. MS has this habit of referring back a lot of things to "needs WMP to run".

[superboyac]

That vimeo video is really incredible.  I'd be curious how difficult it was to get it to look like that, and what kind of equipment was used.  Looks amazing.

[JavaJones]

Requiring WMP is pretty lame! Has anyone tried the mobile apps yet?

- Oshyan

[4wd]

http://bgb.4x4aus.net

Top video has an audio track so was timelapsed to match the length of it, considering the footage was recorded on a handheld MPEG2 camcorder, (ie. the original is very stability challenged), the Hyperlapse version turned out very smooth.

In case anyone is interested, the data on the bottom of the top video comes from a GPS data logger, (you can see it hanging from the mirror at around 1:20 - easier in the Hyperlapse version), which I exported in GPX format and then converted to a .srt file to mux into the video.

The track is called Billy Goat Bluff Track, it drops ~1200m over ~7km - unfortunately the camera battery went flat about 500m from the end of it.

[tomos]

^ I enjoyed that drive
was impressive to see how quickly the elevation changed (I know it speeded up, but still).
And very apt music :-)

[JavaJones]

Cool trek! Now I know why you're called 4wd. And not a bad result on the hyperlapse either.

- Oshyan

[4wd]

^ I enjoyed that drive

-tomos (May 19, 2015, 04:53 AM)

Thanks, now that I've got a camera that can do FHD I'm thinking I might have to do it all again 

Unfortunately the various park authorities have closed/repositioned the more interesting tracks since I started 4wding more than 30 years ago.

And very apt music :-)

It was a choice between B.T.O. or Jethro Tull's 4.W.D.

[tomos]

That vimeo video is really incredible.  I'd be curious how difficult it was to get it to look like that, and what kind of equipment was used.  Looks amazing.

-superboyac (May 18, 2015, 09:22 AM)

I came across this description:

I had to import and customize the NEF files before I
equalized them with the great LR-TIMELAPSE from Gunther Wegner. (http://www.lrtimelapse.com/)
(Adobe Lightroom is necessary)
The observed jpeg had then to be droped into virtual dub and were rendered as avi.
When this was done, I had to stabilize the sequences manually frame by frame (AE motion tracker) and rendered each of them in 3 different sizes: (4928x3264 pixels, 1920x1080 pixels, 1024x768 pixels)
Last but not least the snippets were edited fitting to the beautiful title "Diving Through The Blue" by the respectable composer and musician Valentin Boomes.

from this video -
https://vimeo.com/50238512

Tbh I dont understand half of that description - but it does sound like a lot of work. Surprised the stabilizing has to be done manually. Also not sure what exactly the LR-Timelapse software does.

They seem to generally use full-frame cameras - I guess if you're going to put all that work in, you may as well start with a quality image.

[4wd]

Been playing a bit more with Hyperlapse, (FHD up and down Billy Goat Bluff this time), and while it may produce good results it has some drawbacks, (AFAIAC), compared to increasing the playback rate in a NLE.

- Subtitles can't be burnt in to the source video because they will be effectively removed by Hyperlapse due to the way it works.  This means some tedious editing of external subtitles to get timing correct - the following two points make using external subtitles almost impossible.
- Instead of a constant rate increase you get a rather slow/fast effect, slow where the camera is panning, fast where it isn't - which can be quite annoying sometimes.
- The big one is you can't set a specific duration of the video, you only have a choice of x2-x25 which usually doesn't give the time that it should be.  This makes it very hard to match it up to an audio track.  Vegas I can at least specify how long the video is going to be down to a specific frame.

[superboyac]

That vimeo video is really incredible.  I'd be curious how difficult it was to get it to look like that, and what kind of equipment was used.  Looks amazing.

-superboyac (May 18, 2015, 09:22 AM)

I came across this description:

I had to import and customize the NEF files before I
equalized them with the great LR-TIMELAPSE from Gunther Wegner. (http://www.lrtimelapse.com/)
(Adobe Lightroom is necessary)
The observed jpeg had then to be droped into virtual dub and were rendered as avi.
When this was done, I had to stabilize the sequences manually frame by frame (AE motion tracker) and rendered each of them in 3 different sizes: (4928x3264 pixels, 1920x1080 pixels, 1024x768 pixels)
Last but not least the snippets were edited fitting to the beautiful title "Diving Through The Blue" by the respectable composer and musician Valentin Boomes.

from this video -
https://vimeo.com/50238512

Tbh I dont understand half of that description - but it does sound like a lot of work. Surprised the stabilizing has to be done manually. Also not sure what exactly the LR-Timelapse software does.

They seem to generally use full-frame cameras - I guess if you're going to put all that work in, you may as well start with a quality image.

-tomos (June 18, 2015, 04:13 AM)

boy, it really is a lot of manual work.  i think i was assuming it was more automated.   I don't know what to think now...if there is that much work required, to me it seems like regular video editing.  So now I'm underwhelmed by the hyperlapse software.

[JavaJones]

Uh, perhaps I should note that the term "hyperlapse" is a *general* word, it's not Microsoft's name for the software. The video you link above with all those complicated steps is *not* using Microsoft's software. It is, in fact, a description of *exactly* the kind of in-depth, laborious work that Microsoft's research project was trying to avoid and, to some degree, it succeeds in doing so, although the results are certainly less controllable (but also a helluvalot more automated; 75% or more of those steps are unnecessary with Microsoft's app).

- Oshyan

[MikleB]

G'day 4wd

So that's what Billy Goat Bluff Track looks like. Very interesting... Worked for 10 years with a bloke who regularly did some 4 wheel driving and he has talked about that drive a couple of times as he has done some tracks in the Alpine country. Bit of a photo buff and he has some stunning prints from around that area, mainly Craig's cabin and surrounds.

Just viewed both your videos on the descent you did and comparing them it seems that the Microsoft product looses a noticeable amount of detail and the colours are also a bit washed out but way smoother a view and a better experience from a casual observers point of view another post intimating that you have done a little more research into the results obtainable using a high definition camera and a few comments that I wish to explore a little if I may.

Been playing a bit more with Hyperlapse, (FHD up and down Billy Goat Bluff this time), and while it may produce good results it has some drawbacks, (AFAIAC), compared to increasing the playback rate in a NLE.

- Subtitles can't be burnt in to the source video because they will be effectively removed by Hyperlapse due to the way it works.  This means some tedious editing of external subtitles to get timing correct - the following two points make using external subtitles almost impossible.
- Instead of a constant rate increase you get a rather slow/fast effect, slow where the camera is panning, fast where it isn't - which can be quite annoying sometimes.
- The big one is you can't set a specific duration of the video, you only have a choice of x2-x25 which usually doesn't give the time that it should be.  This makes it very hard to match it up to an audio track.  Vegas I can at least specify how long the video is going to be down to a specific frame.

-4wd (June 18, 2015, 05:57 AM)

 Regarding the subtitles, I presume you had to insert these into your Vegas video after shooting using Vegas itself (I know of the software but am not at all familiar with it.) and that the Microsoft software stripped it out during it's processing. I'm presuming you added the GPS stuff to the full normal speed video then processed the subtitled video using both software packages to get the shortened trips, or did Vegas (Another presumption on my part, it being that you mainly use Vegas and are just comparing it's timelapse capabilities to the output from the Microsoft Hyperlapse software.) add and shorten at the same time?

 Just trying to get a better handle on what I am seeing. If you did indeed need to do two passes, either for the software's sake (Can't add subtitles and speed up playback in the one pass.) or for your own archival needs (Want to keep the full, subtitled, movie but just want to have a quick little episode to show non-4 wheel drivers what you are up to out back {I mean this literally as I'm, just over the hill in Kilsyth  .}  and not bore us. So added subtittles to the full movie and used that version for the shortened version on both occasions.)

 I ask mainly from the viewpoint that this is still software in development it seems and the crowd from Microsoft who are developing it may find experiences like yours very informative as to what they need to make it useable in the real world. They may have considered GPS subtitles but I'm not sure that the Go-Pro cameras have any GPS capabilities and I'm certain that many people who would use their software would use this feature to catalogue events they wish to keep whether Go-Pro's have this feature or not - there are other movie camera brands of course. Yes I'm a lapsed keen photographer and this kind of software is really interesting. I just feel that the earlier in development that these sorts of issues are raised then the better the solutions can be integrated into the final product.

 While watching and listening to your videos I had memories flood back of one earlier time that I first consciously recall this sort of road trip effect and that was the in 'The Hunter's and Collector's - The Slab' music video but that was not time lapse just edited segments, but used to good effect. Sorry for the rant but the Microsoft site states that they have a forum for this software and it really is an intriguing concept to have these sorts of capabilities in the average user's hands and I am trying, not so subtly, to get you to make your experiences known to them so that they may be able to incorporate some sort of secondary sub-processing channel just for subtitles to be kept and in sync. I feel however that it must not just be tacked on at the end as an afterthought.

 However in the mean time, I wonder if you are making two passes, then maybe it would be possible to process via the Microsoft software first and not apply any speeding up but just to see if smoothing can be applied firstly and then add the GPS stuff to this resultant video and then just do a speed up in Vegas and have the capability set a predefined time length. This process has now become a three pass one instead of two but if it's just a matter of starting it up and letting it run for each pass it might presently be acceptable for the odd video or two.

 Hmm, thinking on this, I see at least three process pathways that it might be possible to proceed along to get the some desired results and also that they all have there pro's and con's. How much time is involved, what quality losses occur, can manipulations in one process alleviate problems encountered in the other process if done prior to of alternatively after (I.E: Alternate processing orders.) and just what these might be for any given source video and required result... OK, before I get too bogged down, I must state that I may only look on in wonder as I only use a 32bit Windows XP computer so can't do any of these tests myself but would love to do so (I do have some video gear though, just a little antiquated for the types of trials that I would love to perform. Ah, more expenditure looming.     at some point.)

 Pondering the panning speed up problem this I'm sure could be fixed in some pre-processing scenario, but now it becomes another 'pass' to go through but this time one that maybe only needs to be used to set timing marks as a pre-process, for rectification values if you get my drift, as an analogy, like is needed and done within CGI animation softwares for fly through's (I.E: If nothing new to see, go fast, if lots of new data, slow down but fit it all into the required 4minutes 22 seconds, start soundtrack at beginning, add narration tracks and backing tracks, add opening titles, sub-titles and end credits, drop in a still frame here, etc... And when all is ready, then do it all in one pass, but keep a record of the settings for reapplication if the results need tweaking.). Sorry but, these thoughts come to mind both as problems and solutions/ideas as to what would really be a - usefull - piece of software with lots of power but with simplicity built in (Heaps of core functionality, properly integrated and well laid out.).

 The thing is, IF Microsoft do polish this up and make a final release of it, then I'm sure someone else will bring out a competing product, without any guarantee that it would be a better product or of necessity a worse one... Who has the money to research like Microsoft can do? I feel the best outcome would be if Microsoft were to be given as much input as they could handle so as to be fully informed of any foreseeable requirements presented by the processes we users would wish to perform, then the more likely that their research team will see that the software itself is one that the community at large is keen to have made available to them, albeit improved upon in usefull and sensible ways, and as such will continue it's development from which we might see a piece of very usefull adjunct to home video production in this case. Home presentations to be effective need scripting just like a Hollywood movie, otherwise boredom can and often does set in. Seen a few sad affairs and some good ones too. Hyperlapse technology it seems would do a lot to help out in that area (Wether from Microsoft or not.).

 I hope that you do intend to address some of your concerns on the Microsoft site and hopefully receive feedback from the developers.

 I'll end here as this is enough from me indeed. Looking to see what thoughts you may have regarding these issues or just this piece of software in general from your further use of it.


MikleB

[4wd]

{I mean this literally as I'm, just over the hill in Kilsyth  .}

-MikleB (June 25, 2015, 07:43 AM)

Actually, you're not "just over the hill", you're down the bottom of it - I'm in Kalorama

Regarding the subtitles, I presume you had to insert these into your Vegas video after shooting using Vegas itself ...

The subtitles for that video were burnt into every frame of the original video using either ffmpeg or mencoder, (I forget which - one was definitely easier than the other though), the output being to uncompressed video to avoid recompression artifacts.
The video is then loaded into Vegas and then the playback rate is increased or decreased by holding down Control and dragging the end of the video one way or the other.  On rendering, Vegas (AFAIK) just does frame decimation in order to end up with specified playback length at the specified frames/second.  (BTW, Vegas 11 only allows a maximum 400% rate increase, I had to render, reload, and increase the rate again to get it to ~600%.)

eg.  A 4 minute video at 30fps when increased to a x2 playback rate will have half the total number of frames removed in order to play at 30fps in 2 minutes.

Doing it this way means that the remaining frames are left untouched, so the subtitles are still fine.

What I think MS Hyperlapse is doing (WARNING: simplistic theory following) is it does the frame decimation but then it uses tweening^w to create frames that morph the content between the original frames.  This results in burnt-in subtitles being morphed out of existence.  If you watch the video at First-person Hyperlapse Videos it's very noticable during the rock climb section (~1:55 onwards) where you can see rock formations and the other climber appear to morph between formations/positions.  Note, the default fps setting for MS Hyperlapse is 60fps, this gives them twice the number frames in which to make the result appear smoother.

During panning you have more movement (relatively) than forward motion so the number of inbetween frames created increases so the motion appears smoother and a small time stretch occurs.

I doubt there's anything that can be done about burnt-in subtitles that wouldn't require a massive rewrite to enable text detection within an area, and when that text is changing almost every frame I would think that the effort involved far outweighs the result.

So then, external subtitles - for the basic frame decimation that Vegas uses this isn't a problem as the playback rate is applied linearly across the whole video.
If the rate is set to x2, (assuming 30fps), then 15 frames will removed from each second of the original video and the time between each of the remaining frames will be halved - so the playback time is effectively halved.  Easy to calculate for use in external subtitles.  So easy I wrote two simple programs to do it for me.

With MS Hyperlapse, no such luck - the subjectively different playback speed between areas of low motion and high motion, (where extra frames need to be inserted to give the appearance of smooth motion), means trying to synchronise any external subtitle is a laborious job in text editing.  You'd need to keep referring back to the original video to locate a point where a subtitle should be, locate the same point in the Hyperlapse video, (within milliseconds), and edit the subtitle file with the subtitle start/end time.

If you're not referencing anything in the video, then you can get away with it.  However, for the stuff I'm doing it stands out like dingoes gonads, eg. the altitude reading is still increasing even though the video clearly shows I'm heading down the hill.

As an example, I'll upload a Vegas accelerated and Hyperlapse video some time in the next week or so that shows what using external subtitles looks like in each, (the Hyperlapse subtitles will just be auto-generated, no hand editing).

As for being able to specify a playback time for the end result, I'm not sure that could be done without increasing the number of passes the program needs to do, ie. it would have to run though choosing optimal frames to keep so it could generate a specific number of inbetween frames to end up with a specific playback time.

Finally, it should be noted that the only thing I did in Vegas was apply a rate increase.  I didn't use the Stabilize Media or any Video FX - that'll be my next experiment

[Curt]

[deleted]

[4wd]

Hyperlapse is scheduled for final release on October 01, after that you need to buy a license to export without a watermark.

Current preview version will let you export without a watermark until release date.



From a quick search I can't find a mention of projected cost but I hope it's reasonable, (just picked up a HTC RE for a really good price ).

[JavaJones]

Oh bummer, wonder how much it will cost. I personally think it's a mistake to charge, they'll never make much money on it and they were getting lots of good press about it. Most people who will use it will probably be just home users who probably won't pay... My assumption anyway. We'll see. But they're certainly limiting their potential audience now...

- Oshyan

[4wd]

Hyperlapse is now available for Android: Microsoft Hyperlapse Mobile

[4wd]

Microsoft Hyperlapse Pro trial has ended and the final version is now available from the MS Store at a cost of $49.99

Store link

Microsoft "Hyperlapse" tech is finally available to play with