So after 13 years I’ve left TriMM and Enschede behind to focus more on games and mobile apps. Even more specific, if possible, educational software, aimed at kids. But I’m not that picky tbh. Although I’ve been working for NeuroFlasher and Inner Drive since 2006, it still feels kinda weird, but I’m sure the dust will settle soon.

My big plan was to have the new site up and running before starting, but then the client work started coming in and… there you are still staring at my old blog…

Some further research indicated that it is caused by a bug, through a combination of a dynamically created textfield on top of a clip with a scrollrect set, on which I draw using the drawing API.

The solution is in the CubeView class, instead of setting the scrollrect on the _canvas, set it on the _canvas._parent. I’m not going to update all downloads, but now you know how to fix it .

Although if you need real performance you will probably unroll the 3 loops.

Download the sources here: 3d Panorama v0.7 (287)

With respect to the last post/implementation, only a few minor changes were required to the demo we had so far. Basically making the hotspot light up is implemented by drawing on the plane’s material. That’s all there is to it. I’ve implemented this idea in the interactive material class.

Lots of room for optimization, but the basic principle remains the same.

Although the goal has never been to deliver a set of reusable classes/components, the basics are all there, and the possibilities are legion. And although the architecture of the code samples can be improved, and they are far from being a complete application, most of the principles and a lot of the sources can be reused and applied to your own panorama implementations. Also note that you might need to clean the code a bit (I saw some left over parameters sneaking around that are no longer used, hunt them down!)

So this concludes my panorama series, hope you enjoyed it. Onto the next project!

Below is a screenshot of the final version with real images and hotspots implemented. The image is a 3d image and was rendered by my colleague at TriMM, Sebastiaan Dorgelo.

Download the sources here: 3d Panorama v0.6 (225)

How you define your hotspots is up to you. For example you could write a tool which allows you to edit and generate a CubicPanorama. For a project for the dutch police academy I’m currently working on at TriMM, I am writing such an editor in Flex. Here is a screenshot of the work in progress:

Basically it allows you to select a cube side, import a picture for it and draw hotspots on it. The complete definition/model for the cube is then saved to xml, the polygon information with it.

In this demo I’m simply generating rectangular hotspots on the fly randomly as you can see when you reload the page. No matter whether you are using irregular shapes through xml or random rectangles the principle stays the same.

One important note though before I continue, the hotspots do not have to be visible and not rectangular either. The only reason they are here, is to keep it simple and to provide some visual clue of where you should point and click. In addition whether you show a handcursor or not is all up to your implementation.

But like I said the principle stays the same. In the last post we showed how to derive an x,y coordinate within a plane’s texture. But we do not have to use the plane’s texture, we can use any x,y lookup system. For example an array as a look up table. But wait, there is a better way, we simply use another bitmap as a lookup table.

So the basic idea is: for each cubeside, create another bitmap called lookupbitmap with the same size as the texture, and draw the hotspots for each plane onto the lookupbitmap with the hotspot’s ids as colorvalue.

From an architectural standpoint there are again multiple approaches of course. I introduced the CubicPanoramaModel a couple of examples ago, so if you do define your hotspots in xml, that would be a good place to store them. I decided to leave that as an exercise to the reader (you) however, and keep it simple.
We could implement a bitmap manager, material manager, hotspot manager or whatever. I decided to implement this feature in a subclass of a plane’s material and call it InteractiveMaterial. In this case, since it’s a bit hacked together and unoptimized, it’s called MockupInteractiveMaterial, since it is just there to demonstrate the principles.

The MockupInteractiveMaterial generates 5 random hotspots with an id of 1-5 for plane 0, 11-15 for plane 1, 21-25 for plane 2 etc. It paints these hotspots into it’s lookup table and onto it’s own material so you have a visual clue of where to point your mouse. You might even want to be able to toggle this hotspot visibility in which case you would have to retain a copy of the original material, before you ruin it with random rectangles.

There are a couple of things to keep in mind though:

• you will have to deal with overlapping hotspots. In this example I simply drew the hotspots into the lookup table using the lighten blend mode, which will give the hotspot with the highest id precedence.
• by default flash will draw everything anti-aliased, which cannot be switched off except by setting quality to low. Drawing a hotspot with an id of 10 will result in some pixels around the edges of the hotspot having id 9 which ruins the idea. The ‘trick’ is to draw your hotspot into an inbetween bitmap and threshold that bitmap on the id/pixel value that you want.

Checking which hotspot is under the mouse is now as simple as extending the interactive material with the following method:

Download the source here: 3d Panorama v0.5 (191)

In the previous post, we implemented a vector pointing at the cube plane under your mouse. In this post we will look at deriving the local x,y coordinate within that plane, and with it the local x,y coordinate of the pixel within the plane’s texture under your mouse. (Although for hotspots, any local coordinate system will do).

In order to do so, we need to project the vector at the plane it is pointing at. This results in a coordinate that lies within the plane. The coordinate comes from a set of coordinates that all lie within that plane, and this set can be mapped to a range representing the texture coordinates (or hotspot coordinates as we will see later). Although this might sound complicated, it is not so bad as it sounds.

Let’s review projection first.

The formula for projection was:

px = – (projectionplanedistance / original_z) * original_x
py = – (projectionplanedistance / original_z) * original_y

It doesn’t matter whether the original z lies in front or behind the projection plane.

My next two crappy sketches demonstrate this point, first behind the plane (I have left the y coordinate out of the images for clarity, so it is as if we are viewing the 3d scene from the top):

Our original point lies at an (x,z) of (1.5,-6) (in other words x,y,z = 1.5,0,-6).
Our projectionplane lies at a distance of 4 from the origin:

px = – ( 4 / -6 ) * 1.5) =>
px = 2/3 * 1.5
px = 1

Our projected point has an x of 1, at a z of -e.

A point in front of the projectionplane is handled the same way:

Our original point lies at an (x,z) of (1,-2) (in other words x,y,z = 1,0,-2).
Our projectionplane lies at a distance of 4 from the origin.

px = – ( 4 / -2 ) * 1) =>
px = 2

Our projected point has an x of 1, at a z of -e.

In a cube we can do the same, by projecting each vector at the plane it is pointing at no matter what the length of the original vector was:

The distance e we choose is arbitrary, it merely influences the range of the resulting projected values which have to be normalized/mapped to the texture coordinates anyway. However if we choose 1, it simplifies our projection formulas:

px = – (1 / original_z) * original_x
py = – (1 / original_z) * original_y

which is

px = -(original_x / original_z)
py = -(original_y / original_z)

Which will result in a mapping of the vector to a projected px and py within the range -1 to 1.

BUT which coordinate values we should use as x, y and z’s for our projection formulas depends on the orientation of the plane we are projecting our vector at, take a look at the following image:

The projection formulas are based on the projection of coordinates having a negative z with respect to the plane we are projection points on. So for plane 0 we get what you would expect:

px = -(original_x / original_z)
py = -(original_y / original_z)

And in plane 0, z is negative, so this is the same as:

px = (original_x / Math.abs(original_z))
py = (original_y / Math.abs(original_z))

The only reason I’m writing it that way is that we already calculated the maximum absolute values of x,y,z to determine which plane we were pointing at. We see the original upperleft corner in plane 0 has an x,y value of -1,-1 to a lowerright x,y value of 1,1, so there is nothing extra we have to do to map these coordinates from topleft to lowerright to a range of -1 to 1.

For plane 1, we see that the negative z value we need for our projection formulas is actually a positive x value, and that the local x,y is represented by a pair of y,z coordinates. So we negate our x value and fill in the rest:

px = -(original_y / -original_x)
py = -(original_z / -original_x)

which is

px = (original_y / original_x)
py = (original_z / original_x)

Since the topleft y,z = -1,-1 to bottomright y,z 1,1 is already mapped correctly no further action is required.

(If you are confused about the planes and their corner coordinates, check out the previous post which displays the cube with all it’s coordinates per plane.)

For plane 2, we see the negative required z value is actually a positive z value, and the local x,y pair is actually a local x,y pair, so we get:

px = -(original_x / -original_z)
py = -(original_y / -original_z)

which is

px = (original_x / original_z)
py = (original_y / original_z)

BUT WAIT! We see that the upperleft coordinate of the plane is not -1,-1. It’s 1,-1. And the lowerright is not 1,1 but it is -1,1. In other words the x range has been flipped. To correct this we need to negate the px:

px = -(original_x / original_z)
py = (original_y / original_z)

We can do this for each plane, but I will spare you the pain, since I already did that in the source code.

Now we have the calculations that map our mouse position within a plane to a local x,y coordinate with x and y both in the range -1, 1. We can map this to a texture pixel coordinate. Assuming our projected local x and y are represented by planeX and planeY we get:

texture_x = (plane_x/2) * texture_size + (texture_size/2);

which is the same as:

texture_x = ((plane_x+1) * texture_size) /2;

and ofcourse for y:

texture_y = ((plane_y+1) * texture_size) /2;

The example at the top of the page demonstrates these principles. Check out the updated checkPlane method. Note that the texture size is 400 x 400 pixels.

Finally in our next post we will implement the hotspot detection and then we have only one post to go I think which demonstrates hotspots in a 3d panorama that light up as you mouse over them and are correctly transformed in perspective along with the rest of the cube.

Download the source here: 3d Panorama v0.4 (118)

In order to be able to detect hotspots under the mouse, the first thing we need to do is find some way to detect which plane the mouse is currently over and what the local x,y coordinates of the mousecursor are in the local space of that plane.

Instead of providing all the theoretical background I want to try and make it conceptually clear how we can do this. So let go of the mouse coordinates for a moment and take a look at our untransformed starting cube we discussed in a previous post:

We can see that a cube is nothing more than a space enclosed by 6 planes. We can even specify the equations for these planes:

The plane numbers correspond to how we have defined our planes in the panoramas these past examples. Contained within this cube is the set of all (x,y,z) coordinates with an absolute value for x, y and z less or equal to 1. In addition for the coordinates that lie exactly within a plane of the cube, ’special’ equations hold. For example for plane 0 we could say that Math.abs(x) <= 1 AND Math.abs (y) <= 1, which is the same as saying that for all points in plane 0, Math.abs(x) <= Math.abs(z) AND Math.abs (y) <= Math.abs(z).

But this is true for plane 2 as well, so how do we differentiate between these two planes? Simple, check the sign of z. If z < 0 and Math.abs(x) <= Math.abs(z) and Math.abs (y) <= Math.abs (z), or even shorter, if z < 0 and Math.abs(x) >= z and Math.abs (y) >= z, we are dealing with points in plane 0.

It gets even better, if the planes are closer or further away from the origin (0,0,0) and in effect the cube is smaller or larger, this will still hold. In order words seeing the z instead of -1 and 1 in the example equations above means that these equations hold for planes parallel to our planes as well. And we can write these kind of equations for each plane. So given any point you can tell in which plane (or rather quadrant) the point lies by looking at “the sign of the coordinate having the largest absolute value” (from page 142 from Mathematics for 3d Game Programming and Computer Graphics).

Visually this means that given any point contained in the cube you cannot only tell in which plane it lies, but in which quadrant as well. In other words if you view a cube as 6 pyramids put together, you know in which ‘pyramid’ a point lies by looking at “the sign of the coordinate having the largest absolute value”.

(A cube’s content split into 6 pyramids)

So the next problem presents itself: when we are pointing at a certain pixel in our cube, what is it’s coordinate? We can calculate the coordinate by pointing a vector in the direction of our mouse pointer and see where it intersects one of the 6 planes. But how do we calculate the direction of the vector based on the position of the mouse pointer?

Remember the information we discussed in an earlier post about relating the field of view, wall size and wall distance (projection plane distance)?:

projectionplane_distance = (wallsize/2) / tan (fov_in_radians * 0.5)

Visually:

The area within the rectangle on the wall, is the area of our 3d space actually visible on the screen.

(Note that we are using the same field of vision horizontally as we are vertically, like it has been in our panorama’s all this time)

What this did in effect, was given a wall size (stage size in pixels), and a field of vision, calculate the projection plane distance.

Once we have a projection plane distance, we could pick a point in the visible area of our 3d space, in other words a point on our stage, in other words at a certain distance in pixels from the center of the wall/stage, and calculate what the angle of a vector towards that point on the wall would be:

First for the angle alpha which represents the angle between a vector pointing straight at the screen and the mouse x position:

projectionplane_distance = x_offset_from_center / tan (alpha) =>
projectionplane_distance * tan (alpha) = x_offset_from_center =>
tan (alpha) = x_offset_from_center / projectionplane_distance =>
alpha = atan (x_offset_from_center / projectionplane_distance)

The x_offset_from_center is the difference between the center of the stage and the mouse x in pixels.

Of course the mouse position does not only present an offset across the x axis but across the y axis as well:

We can see the same formula applies, however instead of the projectionplane_distance we need to use pythagoras on the plane distance and the mouse y offset from the center, to calculate the length of our adjacent, which gives us the following y angle:

beta = atan (y_offset_from_center / sqrt (projectionplane_distance^2 + x_offset_from_center^2))

So now, given any vector that goes through our camera and points at the center of the screen, we can create a vector that points at the pixel under our mouse by rotating it by the calculated angles.

rotatedvector = [0 0 z].rotateY (alpha).rotateX(beta)

Visually:

Pick whatever you want for z for example [0 0 -1] or [0 0 -10] or whatever. As long as x and y are zero and we are pointing away from the camera.

So now we have rotated vector, what can we do with it?

If we view our rotated vector as a point, we can determine in what quadrant of the cube we are. However we are faced with another problem. The principle that “the sign of the coordinate having the largest absolute value” determines that quadrant only works for an unrotated cube. And we are constantly rotating our cube as we are looking around in our panorama. So we have to ‘undo’ the rotation of the cube on our rotated vector, which gives us the vector we would have had if we were able to point at the same spot in the unrotated cube and use that one to determine in which quadrant (is quadrant actually the right term or should it be pyramid?) our vector lies:

The example in this post demonstrates that principle. You can look around in our cube, in which I have replaced the shrine images by images containing the plane number, and while you are doing that a textfield displays in which plane the calculations think you are currently moving your mouse.

Next time we will see how we can process this info to find the x and y coordinate local to that plane’s texture, which brings us one step closer to our hotspot implementation.

Download the source here: 3d panorama v0.3 (142)

There is an issue with distributing source code for your demo’s. Besides demonstrating what you want to demonstrate, you give people a look at how you program. But the focus is not how I program, it’s demonstrating the concepts of creating interactive panorama’s with hotspots that light up. This means I have to find some middle ground in which the code is kept to the bare minimum without hardly any framework classes or complex program structures, while keeping the example readable and extensible.

Therefore I decided to open this post with a little information on application structures. Probably one of the hardest parts of programming is rising to/with the correct level of complexity. You want to be prepared for anything, but not over complicate things (KISS).

So what do you choose for your application structure? You can choose one of the many frameworks, or you can roll with your own. And after that, do you apply that framework to any application you write?

Personally I love to use a setup which allows me to refactor fast, so I can keep the application as simple as possible and then go more complex when needed. This might not always be preferable when working with larger teams, but for these tutorials, for me it is a nice way to work.

For example:
super simple concept demonstration => some quick & dirty code on a timeline
simple concept demonstration => single class demo without any clear division of responsibilities
concept demonstration => couple classes, mainapplication and subparts start to take shape
more formal simple single view application => mainapplication, mainmodel, mainview (with possibly other names like we have in this example)
large scale multiview application => mainapplication, startup tasks, mainmodel, mainview, submodels, subviews etc. I could write a book about this but I won’t anytime soon.

In the more simpler variants, models might initialize themselves and they might use simple callbacks (like in this example). In more complex variants, models might be deserialized from xml, but queueloaders and services might take care of loading other required data, and everything might communicate through events instead of callbacks, maintaining some sense of progress and reporting errors when necessary.

If the application gets really complex, I usually employ a main application model, through which all the application parts are accessible. For people familiar with PureMVC’s ApplicationFacade, it shares similarities (although the way I build stuff everything is strongly type-checked and I almost never employ singletons. What I am doing in these examples with arrays without accessor functions is something that would be a nono in large applications). If the application gets even more complex, the application gets divided into sub applications / modules / application cores etc and the process repeats itself.

All in all there is a pretty seamless way to scale up when required, and when you know ahead of time that you will be building a monstrosity there is always the option of scaling up from the very beginning. The main thing is that the complexity of the architecture is usually matched by the complexity of what you want to achieve (warranted by).

So back to these tutorials, I’m trying to find a middle ground between demonstrating the principles, keeping a decent structure which can be extended, not over complicating things, not adding a bunch of library classes to bulk load content etc etc. And some days I succeed better at that than other days .

Just remember the focus is on concepts, not on application structure nor holy code conventions wars. And that most of this is written after 9:30 PM when the gremlins have gone to bed .

All that said, since we will be adding hotspots and light up features in the coming examples, this example prepares the way to add those things later. I’ve scaled up the example’s complexity a bit and replaced the DistortedPlane class by a less general CubicPanoramaPlane class.

All images are now loaded externally, and the example can no longer be run from the Flash IDE since there is no longer a fla required (although it is very simple to create one yourself). Instead you should download the latest FlashDevelop version.

As mentioned before, there is no errorhandling, no progress display and probably lots of other stuff missing. Feel free to add it, but it is not part of what I am trying to demonstrate here.

Download the sources for this panorama (fully documented): 3d Panorama v0.2 (148)

In the previous panorama post I mentioned a number of issues that would have to be fixed. For this demo I used another image from http://www.flickr.com/photos/heiwa4126/, who has a set of amazing images, so it’s really worth to go there and take a look (the other images in the flickr equirectangular group are very good as well btw).

So in no particular order, what did we add/fix in this version?

Clipping
The clipping issue was described in the previous post.

Seams
The seams were a surprise to me. I thought the render process caused the seams, and that I would have to fix it by making the planes slightly overlap. I even implemented it first, so if you ever need it, here it is:

however, that didn’t fix it, it only made things worse. Then I discovered that the seams were caused by the bitmapfill repeat parameter which was set to true, when smoothing was turned on for the planes. If you remember, this setting was switched on to prevent drawing lag when points crossed the diagonal of drawn triangle. Since that will never happen when rendering the inside of a cube with our field of view setting, we can safely turn it off. And poof ! No more seams.

(Fluid) motion
I updated the mouse and key movement and added mass. The slight delay after you press a key is now gone as well. See the render function for more info on this.

Correct rotation
The rotation is correct. In order to implement that, we needed to keep the original points untransformed, and store a transformed copy of those points as well. Using two rotation matrices, one for x and one for the y direction, we can correctly transform these points each render pass (but only if we have moved).

Field of vision

I implemented a somewhat more scientific way of calculating the projection plane distance based on a field of view parameter. Please refer to the sources for more info.

What’s next?

We fixed a number of minor other stuff as well and cleaned the code a bit, so we are slowly getting to where we want to be. In the next version I will refactor the DistortedPlane class into a more specific custom PanoCubeSide class to make the performance even better, although I think that for actionscript 2 it is pretty solid already.

In addition we will look at externalizing the images and the implementation of hotspots in the next version!