Godot – 360 Stereo 3d camera projections

Loving Godot, but I have a requirement that seems currently difficult to impossible to do:

Rendering a 3D Stereoscopic 360 degree view of the scene.

Cubemap

We can do a monocular projection by fixing six square 90-degree cameras, one for each axis, rending them each into a texture. Then use a shader loaded with those 6 source-textures to figure out which pixel should be mapped where in the final output.

That’s the approach taken by Cykrios with his Godot 360 system to emulate many other kinds of lens too. Amazingly fast considering all the intermediate steps, no obvious lack of detail.

You might think that you could just do two of those: left eye top, right eye bottom, bish bosh there’s your stereo 3d map.

But no.

As explained very well in this amazing page by shubhmehta, what you’ll get if you do that is a view from two eyes swiveling in their sockets:

when what we need is a map from a head rotating two eye-cams:

When you look backwards, your eyes are the wrong way round! When you look left your right eye is pointed directly at your left eye.

We definitely need a different kind of projection than just rotation about the origin, it also needs a position translation that’s different for each column of pixels in the output image.

Monsterous Camera Rig.

Okay, we can think of extending the method though. Instead of six cameras pointed each along one axis, we can have one camera for every vertical column of pixels on the output image.

Four thousand and ninety six cameras.

Well, for each eye. So.. Eight thousand one hundred and ninety two cameras!

Well, actually you can’t cover 180 degrees with one camera, so each column will need two, one pointed up at 45 degrees and one down. So… Sixteen thousand three hundred and eighty four cameras!

Monsterous!

It can be built though. I built a mono-scopic one and a 4096-pixel wide image filled from 2048 cameras and it doesn’t even crash Godot.

It takes about five minutes for the script to add though, so you really do think it’s crashed. And then it’s like half a minute a frame.

For a 1/16th size model.

Too slow by far. I don’t mind a minute per frame in a batch-render but this is not gonna come close to that, even with a nearly empty scene.

And also the resulting render is only good for maybe half the frame. The rest just streaks, I guess memory issues?

So that ain’t gonna work.

Would have been surprised if it did.

Edit Godot!?

I looked at the Godot source-code. It builds pretty easily.

Camera projection code seems to be mostly in

scene/3d/camera_3d.cpp

And there’s a function there called project_ray_origin that seems like it could maybe be altered to give that 360 view?

Though that’d still just be a socket-spinning projection through 360 degrees, not a head-turning projection.

I changed all these files to try and invent a new camera projection mode PROJECTION_PANORAMA, just as a copy/paste from the “perspective” functions for now, to maybe do some experiments on it at least.

    modified:   core/math/projection.cpp
    modified:   core/math/projection.h
    modified:   doc/classes/Camera3D.xml
    modified:   doc/classes/RenderingServer.xml
    modified:   editor/plugins/gizmos/camera_3d_gizmo_plugin.cpp
    modified:   editor/plugins/node_3d_editor_plugin.cpp
    modified:   scene/3d/camera_3d.cpp
    modified:   scene/3d/camera_3d.h
    modified:   scene/main/viewport.h
    modified:   servers/rendering/renderer_scene_cull.cpp
    modified:   servers/rendering/renderer_scene_cull.h
    modified:   servers/rendering/rendering_method.h
    modified:   servers/rendering/rendering_server_default.h
    modified:   servers/rendering_server.cpp
    modified:   servers/rendering_server.h

And while the new projection mode’s name does show up in the editor’s panel for the camera, pressing it makes no actual change.

I guess I missed some place that needs to know about the extended range of the ProjectionType enum?

I’m very lost, in a very big codebase, and seems like there’s a good chance it’d need more than just changes to the project_ray functions anyway because the camera-origin needs to be translated differently for each screen-X, as well as rotated.

So not even sure it’d be possible even if I got it to notice my new projection mode.

Forks

BastiaanOlij did a few posts in 2016 about the topic and code for V3 but it’s not clear if that
really went anywhere and I can’t tell if their fork still has any changes like that or what other things
it might have.

I saw someone with a system to add a 4×4 transform matrix to the camera’s projection properties but I don’t know enough to know if the kind of mapping needed will fit into a single
transform matrix or not.

Or indeed if that code is ever going to make it into main?

Questions

Is there anyone working on enabling that kind of camera projection with a plugin or a patch or
something?

If not, is there somewhere I can chip in on a bounty?

If not, is there anyone who can at least reassure me it’s actually possible to do if I keep bashing
my head against the Godot code trying to add a new projection mode myself?

Or if I just chill for a few months is it coming in like version 4.5 anyway?

Or will that 4×4 transform matrix actually let you do it? I suspect it’d have to be a different matrix for each pixel column but I don’t really understand how the matrix multiplication magic works.

Image

EDIT: Success

Yes, with help from the forum I figured out a way: Here’s the code on Github