This post is brought to you by the letters ‘K’ & ‘J’

Another nice application of a slitscan process, this time using fonts can be seen at :http://www.c71123.com/typography/slitscan-type-project/

This approach uses Adobe Illustrator CS or CS2, and a script provided by the project’s author. Tré cool.

The outcome of all this testing and tweaking

I produced another image using the video slicing technique. The input footage was a slowly panning shot taken at the National Museum of Australia. I have more of these to process, but the files are not available to me at the moment. I think some of the other footage will produce better images.

Workflow for slice and dice

OK, I gotta say it: “iMovie is crap”. Basically all the problems I had with video transport disappeared when I converted the DV footage to a suitable Quicktime supported codec using MPEG Streamclip.

So the workflow on Mac OS X is like this:

• Obtain footage in DV format or from another source (MPEG, AVI or whatever). , and iMovie is OK for getting the DV footage off a camera, but don’t use the ’share’ feature to convert it to some other format.
• Grab the DV footage if you have to. If it was downloaded to the computer by iMovie, find the movie project, select it and use the ‘Show package contents’ of OS X (see the ‘cog’ dropdown menu on the enclosing folder’s window or control-click and use the contextual menu item of the same name). You will then see a folder that contains the .dv files.
• Convert the DV file (or whatever format you have), using MPEG Streamclip. It’s available from www.squared5.com/ and various other download sites on the internet. I found the Apple Motion JPEG A format that presumably is part of the Quicktime installation to be reasonable compromise for speed, file size and quality. The A and B formats are apparently a legacy from the codec’s earlier days, and now it shouldn’t matter which of the two are used. At 85% quality which is supposed to be ‘visually lossless’ the file should compress to around 60% of the DV size, but I guess the results may vary depending on the source material.
• So after this you should have a .mov file that can be further processed in processing. For my project I use a slicing technique that is basically a slitscan, and it can be run interactively or to pre-recorded points in order to manipulate where the pixe-wide slices of video are taken from.

The problems I was having with iMovie were basically to do with the sync points or ‘interesting time’ markers that it was inserting into converted files. It was putting extra points in which made no sense in the context of where the individual frames began/ended. Also the H.264 format didn’t work correctly with some quicktime.Media object methods, yet I have since tested a H.264 format movie exported by MPEG Streamclip that did not seem to suffer from the same problems so I can only assume that iMovie mings this up too.

I tested a number of different codecs at higher quality settings with a DV file that was around 160 MB. Platform was my ancient 450MHz Blue and White G3 Mac running OS X 10.3.9 and Quicktime 7.1.6. The original file specs:

• DV – PAL, 720 × 576, 4:3, 25 fps, 28.80 Mbps
  Audio Tracks: DV stereo, 48 kHz, 28.80 Mbps
  Frame rate in Quicktime Player ~8 fps actual.

The exported files had no sound exported with them as it was not required. I would expect the frame rate improved (in some cases) because of this. Performance in Quicktime Player was as follows:

• Apple DV PAL
  Frame rate in Quicktime Player ~18 fps actual.
• Apple Motion JPEG A
  Frame rate in Quicktime Player ~15 fps actual
• MPEG4
  Unusably slow frame rate in Quicktime Player
• Sorenson 3
  Unusably slow frame rate in Quicktime Player
• 3ivxD4
  Unusably slow frame rate in Quicktime Player

A bit of math…

I’m using one of the Panasonic DV cameras (model ag-dvc30e) to record some video. Seeing as I would like to come up with a rough guide to camera panning (and perhaps movement through space), I needed to figure some stuff out.

The Panasonic’s digital multiplier is ~9.6. This figure was not quoted directly in any info I could find but was calculated from the 35mm equivalent figures.

The reciprocal of 9.6 is:
1/9.6 = 0.1041667

Which gives a CCD element size of:
0.1041667 * 36mm = 3.75mm
0.1041667 * 24mm = 2.5mm
The 36mm and 24mm above are 35mm film dimensions.

FOV (rectilinear) = 2 * arctan(frame size/(focal length * 2))
(source http://photo.net/learn/fov/)

So by substitution we have for the minimum zoom setting:
HFOV = 2 * arctan(3.75/(4.1 * 2)) ~= 49.15 degrees
VFOV = 2 * arctan(2.5/(4.1 * 2)) ~= 33.91 degrees

And for the maximum zoom setting:
HFOV = 2 * arctan(3.75/(65.6 * 2)) ~= 3.27 degrees
VFOV = 2 * arctan(2.5/(65.6 * 2)) ~= 1.09 degrees
The focal length figures for 4.1 and 65.6mm are taken from the tech spec sheet for the camera found online somewhere.

These figures actually seem accurate despite the guesswork and approximations. The horizontal FOV matched up pretty well when I checked by eyeballing the FOV on the camera’s viewscreen against lines of sight taken from a protractor.

784 pixels horizontally per frame.
~16pixels per degree.
~5760 pixels for 360 degrees.
So 5760 pixels divided by 25 fps gives ~230 seconds for a 360 degree horizontal pan moving at a rate of one pixel per frame.

The original pan I took was around 36 seconds in duration, and 230 / 36 ~= 6.39 pixels per frame horizontal movement. I’m pretty sure this matches up closely to the average number of slices taken when tracking the horizontal movement of that clip (found when I was doing some variable width slice experiments), so the figure of 230 seconds must be a pretty good guide for how long a 360 degree pan should take.

That’s kind of a long time, and a small movement rate to pan at manually, but it’s not too bad with a tripod, practice and planning. Motion control would be nice too though.

Out of memergy error

OK, the Java Virtual Machine doesn’t like it when you try to use more memory than processing has allocated.

A compressed movie may be a few MB or less, but when it is made in to a video cube the size can grow massively. For example a 2.4MB MotionJPEG movie takes up nearly 160MB as a serialized object. Once the size of the videocube object exceeds the memory allocation things break and out of memory errors ensue.

The lab G5’s have 256MB of memory and processing can be allocated all of that, but it’s not really going to cut it with even a 5MB movie. Presumably giving processing all the memory will result in page swapping because the OS needs something too, so performance will not be so great either.

I guess this is just a limitation of the videocube idea. Not sure how to deal with it.

Assesment ideas

Well time is drawing near ot wrap this stuff up. This post is just a quick draft of what I’ll try to get together for assessment.

• The PMovie library for processing – will need to get it in order by adding a few more methods and cleaning things up, putting a bit more documentation together for it.
• The other classes I wrote for processing may need a bit of attention. These Classes should form part of my assessment: VideoCube, VideoTracking, and Serialize (maybe – this is just a general utility really).
• The videocube work that runs vectors through the cube. More variations and finished stills/movies/sketches required for this one.
• Video slicing (or slitscan) works. Will need to shoot more video for these to obtain suitable footage with steady and very slow camera movement. Having tried taking slices greater than a few pixels wide, I think I will settle with the 1 or 2 pixel wide slices that I started out my experiments with. Again the assessment should include number of stills/movies/sketches of high quality.
• The work with supershapes hasn’t really progressed much from a basic implementation. If time allows I would like to try a recursive application, and maybe a hybrid use of it with a videocube or slicing techniques.
• I will produce a ‘written’ synopsis of the video techniques I have encountered in the course of my research this semester. Basically I will either write a series of blogs or a series of webpages describing some works and approaches. I’m favouring the latter idea of webpages at the moment because it is a little easier to include samples, applets, links etc. and the control over the design and format is much better.
• Very much doubt that a 360 degree video application will be done at all. I did some research, but have not made a rig, shot footage, or made anything to de-warp the footage to a rectilinear or cylindrical projection.

Hopefully this will be enough material. Not much of it is at present ready for submission, and I need to inform Mitchell of my submission date. Probably late in the second week of exams if that’s not pushing it.

Interpolation

If I do try to create some larger images from the video slicing and video cube experiments, they will need to be interpolated in order to scale them up for printing or simply larger on-screen display.

I have Photoshop, which supports bicubic as it’s best interp algorithm, but there are a number of other solutions out there. The ‘industry standard’ appears to be the Genuine Fractals plugin which can be used in Photoshop (and compatible image manipulation programs). There are other plugin solutions too. A number of stand-alone software applications also have interp capabilities. There are a reasonable number of interp algorithms available, but comparing them is kind of hard because different methods may be suitable for images with differing content.

I have found a few pages which have attempted to do this:

http://www.general-cathexis.com/interpolation.html
http://www.americaswonderlands.com/digital_photo_interpolation.htm

Unfortunately most of the stand-alone applications are Windows only which is not so great for Mac users who have not yet moved to Intel based Macs running bootcamp, parallels, VMWare, Wine or CrossOver.

More of the same

OK have fixed up the variable width video slicer again (see previous post comment).

The code supports averaging of ‘passes’. Basically to use the thing you must view the movie and record values for how far the features in the scene move by clicking on their position in different frames of the movie. It is possible to do this a number of times and then average the result for each frame, which should theoretically produce a more accurate set of values. The following image used 2 passes:

Maybe it looks a little better than just 1 pass. Maybe it’s the placebo effect.

I’m thinking of giving up on this approach because there are 2 specific problems with it:

• The y movement is not accounted for, so slices often do not join up quite so well at their edges.
• The exposure between frames can change enough to show up obvious joins between slices regardless of how well matched the slices are in other respects.

I think I can code to fix the first problem, by either using the tracking info for the y values (it’s supported by the VideoTracking class already), or by processing the resultant image later on to shift the slices around until they match (I have an approach worked out for this too). But that still leaves the exposure problem.

I was quite pleased with the original single pixel image, so it may be better to just use a tripod and a slow panning rate to generate these kinds of images. I will have to shoot some more video to try that out.

First variable slice test

I have ironed out most of the issues and bugs with the video slicing code in processing, so now I can actually start using it to see what can be produced. This is the first image:

You can see the very obvious slices taken. They seem to mostly be around 20 pixels wide, so the camera is probably moving far too quickly. Also because it is hand-held footage the slices aren’t lining up too well either. Perhaps it’s worth taking a much slower pan using a tripod. I guess this is why Daniel Crookes uses a motion control rig.

For the moment I’ll try taking a few shorter passes where I’ll track the motion of features in a scene on a frame-by-frame basis to see if this gives any better results.

Video slicing coming along

Just a quick post about this processing based project’s progress.

I have managed to get the video transport behaving itself quite well now. It supports normal playing, frame-by-frame in both directions, variable speed in both directions, and has the ability to pause or reset to the start point.

The direction issue has been partly solved. If a movie is set to play backwards, then the controls are reversed, and it sets the start point to the last frame of the movie. The main issue still to fix is the looping behaviour – it loops OK when going from the last frame through to the first, but not back the other way. This isn’t a major problem for use generally so it’s not a high priority to fix at the moment.

Accumulating variable width slices seems to work most of the time (errors pop up from time to time, so there’s still a few things to iron out). I have decided to accumulate the slices into a single PImage file which is then saved out at the end to a png format image. The image size is worked out by summing all the variable slice widths together before putting any slices into it. The image size can exceed the sketch size – I don’t know if there is some sort of implementation dependent limit to this, but I guess it may be worth looking into just in case I try to do something really huge.

The final major problem I have is the quicktime movie format. I tried h264 and it was found lacking. Some other formats just seem to produce huge files. I have had a shot at using mpeg4, but it seems to crash the applet sometimes, and is kinda slow anyway. So the search continues for a better codec to use that has good performance, reasonable file sizes, and cooperates with the quicktime java api. If I can’t find something suitable, then I’ll have to support entering the Delta X values with a low res version of the source movie, and then scale them up to grab the slices from the full resolution version. Not too hard, just more code and a different workflow.