Netcode GameDev

A Universal Netcode Framework

Jason Eaton

Jun 10, 2021

The goal of this article is to establish a system of network code that can be reused for nearly any type of game that is session based. The motivation for this work is to make the creation of networked games more accessible to hobbyists and indie game developers. I hope to accomplish this by developing a methodology that allows you to design your game nearly as if you were designing a local multiplayer game. If you browse hobbyist and indie games on the many platforms that are out there, you will see thousands of interesting games, but you will rarely see any that are multiplayer across the internet. However, when you do see multiplayer games, they are all local multiplayer or sometimes playable across a local area network. It is pretty common knowledge that if you want to make a game that is playable across the internet, the netcode needs to be hashed out from the beginning which can be a lot of tricky work. Many college students get intimidated by how difficult it can be to design netcode that is free of bugs, and even AAA titles can have many strange artifacts due to sloppy netcode. When focusing on local multiplayer, however, it is pretty much just as easy as making a single player game: use an array of player input rather than one player’s input. It is a very simple architecture. Gather every player’s input. Run a frame of game simulation using that input. Render results. Rinse and repeat. Wouldn’t it be nice if you could get your internet multiplayer up and running within the first few days using the process above, then use the rest of your energy simply making a great game?

Well you can, by using the oldest multiplayer netcode trick in the book. Run deterministic simulations on all clients, and simply use the network to relay the input array to all players. Originally all multiplayer games used this framework across a local area network. This worked just fine until the network called heat.net provided a service in the 90’s that allowed you to play these same games across the internet without any modification to them. The biggest problem with this service was that there was a noticeable delay between entering your input and seeing your character make any kind of movement. This is because the latency of the internet is much higher than a local area network. The simulation can’t advance and provide the correct results until you receive all of the input from all the players. Nearly all real time strategy games also use this method, but the delay is usually unnoticeable in that context. Now this issue of latency caused the entire industry to move into a client-server model from then on, which creates a tricky intimacy between your gameplay and netcode. There was another solution to this problem that I believe was overlooked, until around 2009 when the GGPO (Good Game Peace Out) SDK was developed. This SDK is significant because it was created for the most twitch based type of game out there: fighting games. This system removes the noticeable delay by using the local input for the local player, predicting the input of remote players, and rolling back the simulation to make corrections for the actual players input once it arrives. This allows the local player to have the illusion of zero lag for his own movements, and only a loss of 100ms or so in perceiving actions of remote players. Now this works well for fighting games because there really isn’t much state to be rewound, and you can run dozens of frames for correction in a single 1/60th of a second frame. Now what if you wanted to extend this technique to a more complicated game with 1000’s objects being updated every frame, and 64 players?

I have done this in my demo with a “fast forward” system rather than a “rewind” system. The trick is to “fast forward” the local game using guessed input from remote players, and the known input from the local player. This method allows you to only need to store only a single version of the master game state, that is never advanced until the known input is received, and taking a subset of that state to “guess” where everything is going to be in the next 100ms or so. So now the main game loop is going to look like this: Local player enters input. Update master simulation as far as possible using input from the network. Fast forward a copy of the master state as many frames as needed to get to the local player’s current frame using the local input queue and guessed remote input. Render fast forwarded state. Each fast forward is begun from the master state to correct mispredictions. Now the trick to making this efficient is to only fast forward what is going to be affecting what is on screen. This means if you have a large map with 1000’s of objects, you may only need to fast forward maybe 100 of them for about 3 frames or so. This is way more efficient than rewinding all of the 1000’s of objects and resimulatiing them all.

Technique

So the key to making this all happen is to completely isolate your game state. That means you will not be including any libraries from anywhere, not even math. The reason for this is that the libraries need to be deterministic and stateless. This will also make your game code very portable, and you could plug it into a variety of frameworks. In the future the hope is that libraries for games will be designed like this and we can start with something rather than starting from scratch. Here is an overview of everything that will be going on in this architecture.

architecture diagram

Notice all of the arrows are pointing away from Unity. These are code dependencies, and allow for the replacement of Untiy with other frameworks without changing any of the other libraries. Ideally developers would not need to implement servers at all, and all of the tricky input queue netcode would have packages for Unity, Unreal, FNA, etc. Then the developer would create the game core with deterministic and stateless libraries which would allow them to run on any framework that allowed the languages used to make those libraries. The only framework specific code developers would have to write would be the input encoding/decoding, asset management, and output of rendering and sound effects.

For this demo I have two deterministic and stateless libraries. One is a C# fixed point math library I found online and modified to fit my needs. I went with fixed point to sidestep any floating point difficulties, but floating point math is deterministic, the problem is different compilers can have different implementations of various math functions. Spring RTS Engine uses a special float point math library for it to work. The second is a simple hash table that I used to create a container that is simply a set of hash tables that can be iterated using a filter. This is basically all an entity component system is, if I wrapped my update functions into system objects, it would be a micro ECS. Those two sets of code are found in the ECS and FixedMath folders. The repository for the entire project can be found here. A running demo can be found here. I have no comment system for this article at the moment, but if you have a github account (they are free) you can leave comments and ask questions on the issue tracker of the repository.

The game core itself needs to be implemented as an object that has a five function interface. This interface is key to completely decoupling your game logic code from the netcode. In fact I can think of several relay methods and queue algorithms that could be implemented with this interface. I mention more details in a later section. The game core of the demo is completely contained in the GalacticMarauders folder of the github. The five functions are as follows:

Initialize: A function that initializes your game session with data that is identical for all players, such as map selection, difficulty, world modifiers gotten from the match making server.
SetInput: A function that sets the current player input array, set from the network queues
Update: A function that advances the game state one tick (mine does at 30fps)
FastForward: A function that advances the game state one tick using guesses. A default behavior can be to simply be identical to the Update function.
Copy: A function that copies a subset of another game object. The default can be to copy the entire state of the game session. Ideally it would do this without creating or destroying any objects to avoid garbage collection. I used structs and the array copying functions in C# which I believe would be implemented via the C++ memcpy.

I don’t use a render function because I don’t want my game state to know anything about the method of rendering. I simply allow read only access to Unity to render whatever needs to be rendered (it’s not enforced in this code, be careful!). Now if you were making a local multiplayer game, this architecture is very familiar. Initialize your game state then gather input, simulate, render, repeat. Now if you made sure that this game state was deterministic, and the entire game state could be copied with some kind of deep copy, it can already be networked with no changes! This is just how the GGPO SDK works with fighting games that were not developed to be networked. The arcade games were deterministic, and with the magic of emulators, the machine state could be copied from outside the original game code. And since you have all of the state, the fast forward function can simply be the update function. Now my argument is that if your update function is deterministic and you have successfully isolated your complete game state (which is needed to copy), you can go on about your merry way developing your game, putting off the netcode until later and not have it be a grave mistake! The copy and fast forward functions give you a lot of creativity in optimizing away from the brute force method mentioned above. You could simply write a copy function that copies objects on/near the screen, or write a simpler fast forward function, or both. In the demo, I copy the entire game state because everything is always on the screen, but I actually shut off collision events in the fast forward update to avoid costly collision detection. You will notice in the space invaders demo that there are transparent ghosts of all objects, those ghosts represent the correct master state of the game, and the solid ones are the fast forwarded versions.

More Complex Example

I have a game here written in C++ that I eventually intend to use this system, but haven’t put together the deterministic math and ECS libraries yet. It uses C++’s built in math library, and an ECS I wrote from the standard template library. I used Unity and enscripten to compile it to web assembly so that I can run in a browser. It only took me two days to port to unity this way! This game was built with my own libraries that I have been using for the past 10-20 years. It’s nice to get to write code and not worry about your libraries getting updated under you breaking everything you have done. The itch.io link is here, there is a web assembly version (single player) and a windows downloadable version (local multiplayer). The illustrations give an idea of what the game looks like. There are 1000’s of monkeys running around, in the first image you see whats visible to the player. The second is a zoomed out view so you can see all the monkeys running around out of view. Many of those monkies will not affect anything on the screen for the next 100ms or so. So its safe to simulate just what the player sees or will see in the next 100ms (shown as a white box in image 2). You need to take some things outside the screen that may enter it in the next few frames. Also you must be aware that the screen itself is moving, so this part is a little bit tricky and game specific. Also the architecture in the previous section only used two game states, you could also have a third state that remembers the last guessed state and lerp those objects to the new corrected fast forwarded state. Or you could simply use the Unity objects as the third state and lerp them to the new fast forwarded state to smooth out warping and network jitters.

The first image is the player view, the second is a zoomed out view showing objects that won’t affect the player view within the next 3-5 frames, and the objects in the white box that may have some affect.

Netcode

The demo’s netcode is actually incomplete, as there is not a matchmaking server yet, you have to manually join rooms on my input relay server. What I would do is create an architecture that uses a player discovery and room management server to manage all of the input relay servers. The demo uses an input relay server programmed in golang that is currently running on Heroku. I wanted to make this server as simple as possible, and not have to do anything but relay input. Right now it uses websockets, but the beauty of this design is that the netcode implementation and the gameplay logic are nearly completely decoupled. As long as the master simulations all receive the same remote inputs every frame the games should stay synced. I am using a websockets library I found online for Unity that has a nice layer of abstraction to allow your code to work with webassembly and desktop applications with the dot net framework. Ideally this part of the architecture would only need to be developed once and shared among developers. Also ideally there would already be servers running input relays, my concept was to have a site like itch.io where people could upload their games and automatically get multiplayer for free. The whole point of this methodology is so that you can ignore netcode details except for the copying fast forward functions. For example if your game has the properties laid out in the earlier section, you should be able to write a peer to peer or client server model. You can also change the relay algorithm without affecting the core game code. For example my current algorithm forces the game to advance if input is dropped by lagging players, you could also make the server wait until all input is received to send out the frame (this would make the game pause if players lag though). I also found I could adjust the rewind/delay settings differently for individual players to deal with dropped input. This is very similar to how you configure GGPO. The method of relay and algorithm shouldn’t matter.

Conclusion

So what are the benefits of using this methodology? I think the most beneficial aspect of this method is that you can develop your game like it was a single player or local multiplayer and add internet access later, if you keep three things in mind: the update function needs to be deterministic, the state needs to be isolated to be copied, and being aware that reaction timings need to be adjusted for the remote players actions. There are some good For Honor videos that illustrate this last aspect, but just for example, if there is a player animation that is 300ms, players will only get to see 200ms of that animation. Now that I have a clear idea of this structure, I am able to come up with lots of creative ways to optimize the fast forward functionality for specific games since the fast forward function does not have to be deterministic. This means you can use as much as, or as little as the update code for the prediction as you want! For example one of my demos uses a tile map with destructive tiles, and at first I was thinking of all the kinds of ways I could “patch” guessed destroyed tiles in the fast forward temporarily, but realized that I can just let the blocks be destroyed once the master simulation catches up. This is what happens in the demo with destroyed enemies, and you barely notice they blow up slightly late as the master state catches up. You actually start to get an interesting mesh of strategies that are trackable once you get your game state isolated in your mind. Take the exploding block example. If a block the player is standing on is removed 3 frames late, you don’t start falling until 3 frames later, BUT you fall as if you fell 3 frames in the past, and it all stays synchronized without any extra mental effort!

Note About Unity

Unity appears to be headed in the direction to allow developers to use the strategies described in this article. Their entity component system already has an implemented physics engine that is supposedly deterministic. I have looked into Unity’s ECS, and unfortunately it appears a bit too tricky and convoluted to make sure your game will be deterministic. Many of the ECS frameworks I see out there have this problem, for example one other one I used doesn’t allow you to specify what order the systems are executed. Unity does, but it looked very tricky and confusing how to do it. It also has some frame rate issues, and I don’t know how difficult it would be to copy an entire game state or game state subset. Personally, if I was using C#, I would prefer FNA over Unity. Also one last note about parallel processing, I think you could just use the job system in Unreal to get a similar effect as the Unity multi core ECS. Make sure parallel algorithms might be a bit tricky, but I think you can just sort events by entity hash code to get them to happen in the same order and make events single threaded. If I continue to work on this, I’m planning on just using C++ with Unreal since I ran into some problems with the compiled web assembly with Untiy. For some reason when I run the game in Unity, my platforming demo works just fine, then after compiling to web assembly, the characters fall though the floor! It may have something to do with the loading of JSON files, but I haven’t tracked it down yet. It is a bit surprising since my other two demo worked so well! My plan to fix this would be to make some debugging utilities that do state dumps for every frame so I can step though each state change and each frame tick.