H264 movie format

Having stuffed around for a bit with using h264 encoded movies in my latest processing experiment, I’ve decided to use some other format such as mpeg4.

The problem is that some methods in the quicktime Media class don’t return meaningful values or cause the applet to freeze when used with h264 encoded movies.

This would be OK if I was prepared to modify the PMovie class to not use the quicktime Media class for h264 encoded movies, or to provide alternate methods to be used with h264 movies. But I have also since found out that some methods of calculating movie positions as a time in seconds (and fractions of seconds) don’t seem to be accurate.

If some inaccuracy introduces a small error in tracking the position in a movie, and it keeps getting introduced then a small error becomes larger as time goes on, and then it can start causing slightly odd things to happen.

The upshot of all this and perhaps some other inaccuracies that I haven’t gotten to the bottom of is that I no longer trust a cumulative time-based method of going through a movie frame by frame. I need to either make it based solely upon the frame numbers retrieved from the quicktime Media object, or include some mechanism to recalibrate the time based upon the frame count. This second method should produce a more accurate time by simply setting the current position in the movie in seconds to the frame number / FPS.

Ain’t no way to delay that trouble comin’ every day

What I thought was a problem with my oldschool G3 450 turns out to be something else. Having tried to use the newly modified PMovie class with h264 encoded movies at uni I see the same (lack of) behaviour.

I think the problem is with these methods from the QT Media class when used with h264 encoded movies:

media.getTimeScale()
media.getDuration()
media.timeToSampleNum(currTime).sampleNum

It seems that these work just fine though:

movie.getDuration()
movie.getTimeScale()

So use of the first 2 Media methods can be replaced with the Movie versions of them in the PMovie class. That just leaves the problem of determining the current frame number (which is what media.timeToSampleNum(currTime).sampleNum represents).

Seems odd though that these classes should come from the QT7 installation which supports the h264 codec. Perhaps it’s some issue with the Java version number? I have Java 1.4.2 at home, and the uni G5’s have 1.5.0_07 (which is pretty much current, so maybe it’s not that after all…).

Think I’ll just find a different way to get the current frame number. Shouldn’t be too hard to calculate it from the time, timescale and duration info available from the QT Movie class.

And here’s the code

Thought wordpress had a post length limit, but it turns out that if you have the left arrow symbol in your code (say if you use a less-than-or-equal-to comparison operator), then it interprets that as a tag and stops displaying any text from that point on. So I’m gonna try to escape it and see if that works.

/*this is an attempt to output hi-res vector images of
an old recursion sketch that I made for new media 2
I have tried to get better precision by using doubles
instead of float where possible, but some mis-match still
shows up where it would be nicer if some circles
were rendered directly on top of each other where they almost
but not quite match up… It’s probably more to do with
the scale factors too which I haven’t yet ant-fucked.
*/

import processing.pdf.*;

double x, y, radius, theta;
int sqSz = 400;
char strokeColour, fillColour, strokeAlpha, fillAlpha;
color bgColour;
int rotationFactor, presetLvl;
double[] scaleFactor = {0.5, 0.52, 0.5, 0.48, 0.43, 0.4, 0.375, 0.34};

void setup()
{
size(sqSz, sqSz, PDF, “circle_recursion.pdf”);
strokeColour = 0;//0 = black, 255 = white
strokeAlpha = 60;//0 = transparent, 255 = opaque
fillColour = 255;
fillAlpha = 255;
bgColour = color(255, 152, 0);
stroke(strokeColour, strokeAlpha);
strokeWeight(0.5);
fill(fillColour, fillAlpha);
//comment out these to quickly turn stroke and fill on/off:
//noStroke();
noFill();

x = width/2;
y = height/2;

radius = (width <= height) ? width/4 : height/4;
rotationFactor = 6;
presetLvl = 6;
}

void draw()
{
background(bgColour);//refreshes the window to background colour
//more rendering statements go here

theta = -PI/2;//affects starting posn and direction of drawing sequence

//draw the first ellipse (non-recursively… obviously)
ellipse((float)x, (float)y, (float)radius * 2, (float)radius * 2);

//call the recursive circle drawing fn and stop looping
recursCircles(x, y, radius * 2, 1);
noLoop();
println(“finished”);
exit();
}

void recursCircles(double x, double y, double radius, int currentLvl)
{
double circleSize;
//stopping case
if(presetLvl == currentLvl)
{
return;
}

//use the custom scale factors for rotations below 7
//and the level 7 scale factor for 7 axis rtations and above
if(rotationFactor <= 7)
circleSize = radius * scaleFactor[(rotationFactor - 1)];
else
circleSize = radius * scaleFactor[7];
radius *= 0.5;
//recursive case
for(int i = 1; i <= rotationFactor; i++)
{
ellipse((float)(x + radius * Math.cos(theta)), (float)(y + radius * Math.sin(theta)), (float)circleSize, (float)circleSize);
theta += PI*2/rotationFactor;
recursCircles(x + radius * Math.cos(theta), y + radius * Math.sin(theta), circleSize, currentLvl + 1);
}
}

Revisiting something old

I’m going to have a quick look at something from new media 2. The recursion applets I made which can be viewed one my old nm2 site should be able to be output to PDF format for hi-res printing. Shouldn’t take long…

…about 60 minutes later, and here it is:

circle recursion PDF eg

I’m sure these types of images will make an image-setter keel over and die. Can’t wait.

Phallic destruction sequence

The image above shows slices taken from a movie where the camera doesn’t move significantly. It’s an interesting effect using an arbitrary slice width although it’s not really the type of source movie the system is being built for, but whatever… I like this image and that’s that.

Anyway, I’m making some progress toward a system that will allow the amount of change (in a 2D spatial sense) in a series of video frames to be manually tracked. This will allow a video slicing system to determine how wide a slice needs to be to capture closer to a 1:1 representation between the x dimension of a still image composed of slices, and the movement of features across a period of time captured on a series of video frames. The slice width can then be altered to suit each frame as the slices are taken.

This can compensate for video footage that has panned or tracked over a scene at a higher than optimal speed. Still images built from slices of such footage show a distortion where features in the scene are compressed along the axis of camera movement. The distortion can at times be interesting, but higher rates of camera movement and the consequent image distortion can result in a disjointed series of slices.

So far the system allows the user to identify how far a feature in the scene moves over a series of frames. This generates an average number of pixels that aproximates the movement of all features in the scene over the frames considered. This average number is stored in a way that matches each frame in the movie with a ‘deltaT’ value for x and y movement (I’ll only be using the x value for the moment though). This process is the ‘passing’ phase, and it allows (and requires) a user to attach deltaT values to every frame in a series of video frames so that a 2D image can be constructed from slices of those video frames (the slicing phase).

At the moment the passing phase code is mostly complete, but needs some code to serialize (save) the VideoTracking data structure where the deltaT values are stored. The slicing code does not yet use the deltaT values, so it needs to be modified to use that data. It should be fairly straight forward to do this, but there is a hitch… Slices are accumulated left to right which requires video footage where the features move right to left. If the footage runs the other way then at present I run the footage backwards which makes the use of deltaT values a bit trickier. It would be better to swap the slicing direction in these cases, but this is tricky too. So I need to figure at least one of these options out.

Slicing to make panoramas

I shot some video at the national museum a few weeks ago, and have just started having a bit of a play with it using a slicing (or slitscan) technique. It seems that the result is really dependent on getting smooth and slow coverage of the scene captured to video. There’s a relationship between points on objects and screen pixels that I’ll try to pin down more precisely soon. Basically if a point on an object matches up to a screen pixel in a frame, then on each subsequent frame it ideally moves the equivalent of 1 screen pixel with whatever camera panning/tracking occured at that point in the footage. Variations in this relationship results in stretching or compression of features in the scene across the axis of movement.

Here’s a sample panorama taken with a hand-held pan around the ‘courtyard’ of the national museum:

In this image you can see the compression along the x-axis that occurs when the camera pan was a little too fast. This can be compensated for by scaling the resultant image in photoshop on the y-axis:

Photoshop can also stretch the image along the x-axis (same effect in a bigger image), but this results in some deterioration of the image by the creation of new pixels with resampling/interpolation.

Resolution issues

For the image generated by running scan lines in 3D through the videocube, the resolution in the x-dimension for output is not a problem – there are plenty of pixels available for that. I edited a quick test image in Photoshop that was scaled in the y-dimension (but not the x). This smooths out some of the misalignments where each pass joins up.

The image get’s a bit blurry though, but maybe when printed it could look ok. I’m contemplating printing these kinds of long images from a roll of paper to go beyond the standard sheet sizes if I have to.

As for the animation option, that’s already in the processing code I’ve written by default – each pass is one call to the draw() method in processing. The output on screen is an x*t image with a duaration of y frames.

I’m not sure how far I’ll go with this particular experiment, as I have other cats to skin, but the ability to create a long image and looking at how to deal with the resolution is useful for the other things I have in mind.

That last post was cactus…

Despite the interesting effect created by the coding I made for the videocube experiment in the previous post, it was in fact ‘wrong’. I.e. it was not a correct implementation of the concept I had devised. I’ve fixed this now and it’s looking different.

I’m going to attempt to write a bunch of slices out to a file to create a reeeeaaaaalllllllllyyyyy long still picture of the slices that this generates. For example the image above which was pieced together in photoshop using screen grabs is taken from only four passes through the draw command in processing. It takes the same number of passes as the videocube has in the y-direction to do a complete cycle through, so for this image that would be 100 passes.

Another vid-cube experiment

This one takes a bit to get your head around it. Imagine a cube of video. Imagine a line drawn through the cube from the top-front-left corner to the bottom-back-right corner. Sample each pixel that intersects with the line. Draw that sampled slice as a line in the display. Now move the line end points one pixel into the cube in the x direction (so the front goes x++, back goes x–). Repeat until the line end points reach their horizontally opposite corners. The result:

The image is x wide, by t high (t = number of frames in video source).

There are a lot of options for this one considering that the line end points could be situated anywhere and cycled through any number of points in the cube.

More video cube stuff

Coded up a version that shows two views – one going sideways through the cube and the other going top down. Aesthetically the juxtaposition does nothing for me, I thought initially they could go together, but they don’t really join up too well, and they get out of sync anyway.