Translating Screen Effects to a Modern Rendering Pipeline

Background

Long before you could just click a button and make things glow, game developers were finding ways to implement visual effects with the tools available to them at the time.

The N64 Zelda games are no exception. They had various screen effects that would leverage the system frame buffers in different ways. The GPU would write out the frame buffer contents to a location that the CPU could read. Depending on the effect, copies of the frame buffer data would get modified by the CPU to render later by the GPU. A couple examples of these effects include the following:

Color filters

The grayscale and sepia effects in OoT were implemented by taking the current frame buffer and mapping each RGB value to a intensity value, then mixes it with the desired color.

Screen scaling

To achieve the screen scaling effect in MM, the game captures the current frame buffer, renders a black rectangle over the whole screen, then re-draws the captured frame back as a scaled copy centered on the screen.

Motion blur

MM implements motion blur by rendering the previous frame with transparency over the next frame, and repeatedly over a few frames this would appear as a "trailing" motion blur depending on the level of transparency used

Pause menu background

One less obvious example of a frame buffer effect is drawing the world behind the pause menu.

The game stores a copy of the last frame buffer without the HUD before pausing, and then applies an anti-aliasing filter to it. This allows the world to be displayed behind the pause menu without the cost of re-rendering the whole world.

Disclaimer: The descriptions given above are vastly paraphrased. More technical descriptions can be found in the decomp codebase

What we've done in Ship of Harkinian

With Ship of Harkinian, we made two main adjustments to the drawing code for the few effects needed for OoT.

The first was for the grayscale/sepia filters. We added a custom renderer opcode to track grayscale commands by tracking a color value and setting when to use that color value. The grayscale command would be set early in the instruction queue to apply it to everything drawn after it. The color would be applied at the shader level when computing the final color values for a fragment. This means that the filter would apply on the fly, rather than at the end when compared to the original frame buffer handling.

The downside with this is that only one grayscale color is tracked and applied at a time. If later in the render process a new color is temporarily set, this could lead to render instructions after that not having the previous color if it wasn't reset back. This can be seen with using the cosmetic editor to apply custom colors to Link while in the end credits with the sepia filter.

The second was to continue to render the whole world, actors, and effects, while the game is paused. This allowed there to be zero transition when paused as everything is still rendering exactly how it did before pausing.

The main notable downsides to this is that any game logic that is executed as part of the draw methods would continue to run. This leads to some effects like rain/snow, some scrolling textures, Link's hurt flash indicator to "move" or "update" while paused.

These modifications are essentially alternate solutions that don't rely on frame buffer techniques and allowed us to launch SoH without major issues at the time.

What needs to be done for 2ship2harkinian

While the workarounds used in SoH are clever, there are limitations as to how much can be accomplished that way. MM uses more screen effects than OoT, so we decided our best bet would be to try replicate how the original game works by dealing with the frame buffer. If we can do this properly, then we can also revisit the approaches used in Ship of Harkinian to address the aforementioned issues.

The first major hurdle in achieving this is that, unlike the original consoles, the frame buffer data in modern GPUs are not directly accessible by the CPU. Getting the data to be usable means having the GPU copy it to ram location that is accessible to the CPU, performing some modifications to it, then sending it back the GPU, all of which has performance costs.

The second hurdle is that our ports can render at any resolution and aspect ratio. The game code has various buffers that it uses to hold onto the copied data, which are all hardcoded to the consoles original resolution. Additionally, some of the screen effects work by copying a set of "lines" from the buffers, as not all the pixel data can be sent through texture commands due to the TMEM (texture memory) size. Supporting dynamic resolutions means these structures need to grow and shrink to and send all the pixel data for each effect properly.

Thankfully all these effects, with the exception of Pictograph Box pictures, can effectively be implemented primarily on the GPU side through creative use of shaders and holding onto frame buffer data.

A Modern Solution

With the ability to make edits to the game's source code and having control over the renderer, we can start working on the common logic across all the effects.

Create the new buffers

First we need to create additional GPU frame buffers in the renderer, one for each multi-frame effect, and one general purpose/reusable buffer for same-frame effects. For 2ship, this means we need 3 (motion blur, pause menu background, general purpose). These buffers only need to be created once, so we can just hook into the same init logic that the game uses for its original frame buffers, then we just need to track and expose their IDs for other code to reference.

void FB_CreateFramebuffers(void) {
    gPauseFrameBuffer = gfx_create_framebuffer(SCREEN_WIDTH, SCREEN_HEIGHT);
    gBlurFrameBuffer = gfx_create_framebuffer(SCREEN_WIDTH, SCREEN_HEIGHT);
    gReusableFrameBuffer = gfx_create_framebuffer(SCREEN_WIDTH, SCREEN_HEIGHT);
}

Each frame buffer has a backing GPU texture object tied to it which is where our frame data will be stored. Our renderer is already capable of resizing frame buffers whenever the game window size changes.

Copy to the buffers

Then to copy into our new frame buffers we need a custom render command to signal when exactly to perform the copy and what the source/destination buffers should be.

void FB_CopyToFramebuffer(Gfx** gfxp, s32 fb_src, s32 fb_dest, u8 oncePerFrame, u8* hasCopiedPtr) {
    // ...
    gDPCopyFB(gfx++, fb_dest, fb_src, oncePerFrame, hasCopiedPtr);
}

The copy itself requires an implementation for each rendering backend we support (OpenGL, DirectX, Metal). Using OpenGL as an example, we can perform a blit operation to copy the frame buffer objects. The main game frame buffer has additional complexity whenever the port's menu bar is open, as that is also rendered into the frame buffer. So we need to account for this being open by performing a scaled copy of the actual game region. Our destination buffers are single-sampled, so if multi-sampling is enabled for the main game frame buffer, then we first need to resolve to a single-sample buffer of the same size, before performing the scaled copy.

Additionally we track that the copy has been performed into a pointer for the game code to reference later. We can use the value in combination with the oncePerFrame flag so that we only copy the frame buffer once per "game frame", like for motion blur so its effect is the same regardless of the frame interpolation value. The hasCopiedPtr also helps us if there any dropped frames, by telling us the copy command was never executed, so we can react by requesting a new copy (i.e. for the pause menu background being a single frame copy).

Drawing back the buffer

Finally, once we have our copy we will want to render it back at a later point. We can already make the texture ID that is backing each frame buffer be the active texture for rendering fragments. All we need now is a new command to render a rectangle with a specific size that uses the desired texture ID. This command is similar to something like gDPTextureRectangle except we will need to hardcode the tile information and bypass texture loading in the RDP, as our texture is already available in the GPU side to be used.

void FB_DrawFromFramebuffer(Gfx** gfxp, s32 fb, u8 alpha) {
    // ...
    gDPSetTextureImageFB(gfx++, 0, 0, 0, fb);
    gDPImageRectangle(pkt, x0, y0, s0, t0, x1, y1, s1, t1, tile, iw, ih)
}

Bringing it all together

Now that we have the main pieces in place we can begin to tackle the different effects.

For the grayscale/sepia filter we can execute the following commands to do a copy and redraw with a color filter applied. This addresses the original issues in the Ship of Harkinian implementation, as now this is performed at the end of the regular drawing and we turn it off immediately.

FB_CopyToFramebuffer(&gfx, fb_src, fb_dst, false, NULL);
gDPSetGrayscaleColor(gfx++, filterColor.r, filterColor.g, filterColor.b, filterColor.a);
gSPGrayscale(gfx++, true);
FB_DrawFromFramebuffer(&gfx, fb_dst, 255);
gSPGrayscale(gfx++, false);

For motion blur and pause menu background we can follow similar patterns. A request is made to first make a copy, then on subsequent frames we will draw back that copy. There is additional logic that we need for detecting when the game window size changes so that we can take a new copy at the new size.

if (!hasCopied || lastWindowSize != CurrentWindowSize()) {
    status = SETUP;
}
 
if (status == READY) {
    FB_DrawFromFramebuffer(&gfx, fb_dst, 255);
} else {
    FB_CopyToFramebuffer(&gfx, fb_src, fb_dst, true, &hasCopied);
    lastWindowSize = CurrentWindowSize();
    status = READY;
}

Conclusion

Working on 2ship is cool! We've been able to take learned experiences from SoH and improve upon them for 2ship. We hope you've enjoyed this post and we're looking forward to sharing more of the progress and fun stuff we learn!

On an ending note, here are some "behind the scenes" clips/pics taken during the development of these features, enjoy!