Category Archives: Development/RTsoft

Random stuff I’m working on.

PaperCart – Make an Atari 2600 that plays QR codes

What is it about the Atari 2600?

It was the summer of 1983 at Jeff Mccall’s slumber party when I saw my first game console.

Crowded around the small TV we gawked at the thing – an Atari VCS.

The seven of us took turnst. Passing the joystick around like a sacred relic we navigated Pitfall Harry over hazardous lakes, crocodiles and scorpions.

One by one the other kids fell asleep.  Having no need of such mortal frivolity, I played Pitfall all night!

I fainted in the street the next day due to sleep deprivation.  Worth it.

The challenge

It’s kind of mind-blowing that games that originally sold for over $30 ($70+ in 2018 money) can now be completely stored in a QR code on a small piece of paper.

As a poignant visual metaphor for showing my kids how much technology has changed, I decided to create a Raspberry Pi based Atari that accepts “paper carts” of actual Atari 2600 games.

The requirements for my “PaperCart” Atari VCS:

  • Must use the real QR code format, no cheating by tweaking the format into something a standard QR reader couldn’t read
  • 100% of the game data must actually be read from the QR code, no game roms can be stored in the console, no cheating by just doing a look-up or something
  • Runs on a Raspberry Pi + Picamera with all open source software (well, except the game roms…)
  • Can convert rom files to .html QR codes on the command line, sort if need this or we’ll have nothing to print and read later
  • Easy enough to use that a kid can insert and remove the “paper carts” and the games will start and stop like you would expect a console to do
  • Standard HDMI out for the video and audio, USB controller to play

All about QR codes

The QR in QR Code stands for Quick Response.  It’s a kind of 2d barcode that was invented by a Japanese company named Denso Wave in 1994.  They put it into the public domain right from the get go, so it’s used a lot of places in a lot of ways.

QR codes have a secret power – they use something called Reed-Solomon error correction.  It has the amazing ability to fill in missing parts using extra parity data.  More parity data, more missing data can be reconstructed.  Not certain parts, ANY OF THE PARTS.  I know, right?

Reed-Solomon is also used in CDs, DVDS and Blu-ray, that’s why a scratched disc can still work.

Remember those .par files on Usenet you’d use when you were downloading a bunch of stuff in chunks?  Yep, parchives were based on Reed Solomon.

I hid a fun Atari fact in this code.

I’ve encoded some text in the above QR code with error correction set to Level H (High), which means up to 30% can be missing and you can STILL read it!

Go ahead, block some of it with your fingers, put your phone in camera mode and point it at the QR code above.  Does it work?  That’s the Reed-Solomon stuff kicking in.

QR codes automatically jump to larger sizes to encode more data – from version 1 to version 40.

Can you find your way out of this maze?  Does your brain hurt yet? Hope no one took that seriously and actually tried.

Above is a version 40, the most dense version.  My iPhone is able to read this one right off the screen too.  If you have problems, you can try zooming into the page a bit maybe.

This is the first 2900 characters of Alice In Wonderland.  We can store a max of 2,953 full bytes.  A byte is 8 “yes or no” bits.  With that, you can store a number between 0 and 255. 

Because text doesn’t need a whole byte, there are smarter ways to store it which would allow us to pack in much more than we did here –  but let’s ignore that as we’re only interested in binary data.

If I show the QR code too clearly, I might be enabling rom piracy and get in trouble. Weird, right?

This game (Stampede) has 2,048 bytes (2K) of rom data so it easily fits inside a single QR code.

Other Activision classics like Fishing Derby and Freeway are also 2K games but Pitfall! is 4K game.  Using gzip compression saves us nearly 20% but it’s still a bit too big to fit in a single QR code. To work around this I’ve added a “Side B” to the other side of the Pitfall! card.  Cart. Whatever it is.

My paper cart format stores some metadata so the reader can know how many QR codes are needed for the complete game, as well as if the data is for the same game or not by storing a rom hash in each piece.

Emulating a 2600 on a Raspberry Pi 3

I started with latest Retro Pi image and put that on a micro SD card.  RetroPi has an Atari 2600 emulator out of the box that can be directly run from the command line like this:

/opt/retropie/supplementary/runcommand/runcommand.sh 0 _SYS_ atari2600 <rom file data>

So now I just needed to write some software that will monitor what the Pi camera sees, read QR codes, notice when no QR code is present or it has changed and start/stop the emulator as appropriate.

Writing the software – PaperCart

Naturally I chose Proton SDK as the base as it handles most of what I need already.  For the QR reading, I use zbar, and for the webcam reading I use OpenCV and optionally raspicam instead. (no need for OpenCV on the Raspberry Pi linux build)  I put it on github here.

The PaperCart binary can also be used from the command line to create QR codes from rom files. (It uses QR-Code-generator)

RTPaperCart.exe -convert myrom.a26

or on the raspberry:

RTPaperCart -convert myrom.a26

It will generate myrom_1_of_1.html or if multiple QR codes are needed, a myrom_1_of_2.html and myrom_2_of_2.html and so on.  I opened in the web browser, cut and pasted them into photoshop, scaled them down (disable antialiasing!) to the correct size and printed them.

A quick note about zbar and decoding binary data in a QR code

If you want binary data to look exactly as it went in (and who wouldn’t?!), you need to do a little processing on it after zbar returns it with iconv.  Here is that magical function for any future googlers:

string FixQRBinaryDataEncoding(string input)
{
	iconv_t cd = iconv_open("ISO-8859-1", "UTF-8");
	if (cd == (iconv_t)-1)
	{
		LogMsg("iconv_open failed!");
		return "";
	}
	int buffSize = (int)input.length() * 2;

	string output;
	char *pOutputBuf = new char[buffSize]; //plenty of space
	size_t outbytes = buffSize;
	size_t inbytes = input.length();

	char *pOutPtr = pOutputBuf;
	char *pSrcPtr = &input.at(0);

	do {
		if (iconv(cd, RT_ICONV_CAST &pSrcPtr, &inbytes, &pOutPtr, &outbytes) == (size_t)-1)
		{
			LogMsg("iconv failed!");
			SAFE_DELETE_ARRAY(pOutputBuf);
			return "";
		}
	} while (inbytes > 0 && outbytes > 0);
	
	iconv_close(cd);
	int finalOutputByteSize = (int)buffSize-(int)outbytes;
	string temp;
	temp.resize(finalOutputByteSize);
	memcpy((void*)temp.c_str(), pOutputBuf, finalOutputByteSize);
	SAFE_DELETE_ARRAY(pOutputBuf);

	return temp;
}

Want to make your own?

It’s pretty straight forward if you’re comfortable with linux and Raspberry Pi stuff. Here are instructions to set it up and download/compile the necessary software.

(If you really wanted, it’s also possible to do this on Windows, more help on setting up Proton on Windows here, you’d also need to OpenCV libs and Visual Studio in  that case)

  • Write Retropie image to a microSD card, put it in your pi
  • Make sure the Atari 2600 emulator works in Retropie, setup a gamepad or something
  • Enable SSH logins so you can ssh in, I guess you don’t have to, but it makes cutting and pasting instructions below a lot easier

Open a shell to the Raspberry Pi (as user “pi” probably) and install RTPaperCart and the stuff it needs starting with:

sudo apt-get install libiconv-hook-dev libzbar-dev

Now we’ll install and compile raspicam, a lib to control the camera with.

Note: It acts a little weird, possibly because it’s using outdated MMAL stuff? In any case, it works “enough” but some fancier modes like binning didn’t seem to do anything.

cd ~
git clone https://github.com/cedricve/raspicam
cd raspicam;mkdir build;cd build
cmake ..
sudo make install
sudo ldconfig

Before we can build RTPaperCart, we’ll need Proton SDK:

cd ~
git clone https://github.com/SethRobinson/proton.git

Build Proton’s RTPack tool:

cd ~/proton/RTPack/linux
sh linux_compile.sh

Download and build RTPaperCart:

cd ~/proton
git clone https://github.com/SethRobinson/RTPaperCart.git
cd ~/proton/RTPaperCart/linux
sh linux_compile.sh

Build the media for it.  It converts the images to .rttex format.

.rttex is a Proton wrapper for many kinds of images.

cd ~/proton/RTPaperCart/media
sh update_media.sh

Now you’re ready to run the software (note: pkill Emulation Station first if that’s running):

cd ~/proton/RTPaperCart/bin
sh ./run_app.sh

You might see errors if your camera isn’t available. To enable your camera, plug in a USB one or install a Picamera and use “sudo raspi-config” to enable it under “Interfacing options“. (don’t forget to reboot)

If things work, you’ll see what your camera is seeing on your screen and if a QR code is read, the screen should go blank as it shells to run the atari emulator.

You can point your camera at a QR code on the screen and it will probably work, or go the extra mile and print paper versions because they are fun.  You don’t have to laminate them like I did, but that does help them feel more sturdy.

You might need to twist your camera lens to adjust the focus. 

I setup mine to automatically run when the Pi boots (and not Emulation Station) so it works very much like a console.

Running RTPaperCart /? will give a list of optional command line options like this:

-w <capture width>
-h <capture height>
-fps <capture fps>
-backgroundfps <background capture fps
-convert <filename to be converted to a QR code. rtpack and html will be created>

3D Printing the stand

I sort of imagined designing a stylish 2600 themed case with a slot for the paper cart and fully enclosing the Pi, but that would take skill and also require some kind of light inside so the QR could be read. 

So instead I did the minimum – a thing to hold the PI, camera, and easel where you insert the QR code paper.

I used Fusion 360 and designed the stand parametrically so you can fiddle with values to change sizes pretty easily.  The modules are designed to snap together, no screws needed.

You can download the Fusion 360 project here, the download button allows you to choose additional formats too. 

You need to kind of use common sense and print with supports where it looks like you need them.

Conclusion

So that’s great, but you’d like to store Grand Theft Auto 5 as QR codes because they are so convenient?

Let’s see, 70 gigabytes.  No problem. To convert that you’ll just need about 25 million QR codes.  You might want to order some extra ink now.

At one code per paper, the stack would reach a mile and a half into the sky.

If you got this far, you must also be a connoisseur of gaming history and old hardware.  Check these out too then:

They Create Worlds (Podcast on gaming history, no fluff)
Matt Chat (Interviews and info about old games in visual form)
The Retro Hour (Podcast with retro gaming interviews and news)
Atari 5200 Multi-ROM Cartridge Using Raspberry Pi  (Cool, something like this might make it possible to mod a real 2600 to read “paper cartridges”.  Small world, Dr. Scott M Baker wrote BBS stuff too, including Land Of Devastation as well as Door Driver, a utility that allowed a dumb kid like me to write BBS games)

Funeral Quest (the multiplayer funeral parlor simulation) is now downloadable and free

The rumors of Funeral Quest’s death have been greatly… true

It’s been nearly five years since any would-be undertaker has graced its halls.  In 2013 the old XP machine that was running the only FQ server in the world was decommissioned – and that was that, a game like no other silently departed with nary a single youtube video left in remembrance.

I wrote it in 2001 it was to see if a BBS door-like financial structure could make sense in a modern web environment.

I charged $99 (I think?) a year for the enterprising admin (or, a SysOp if you prefer…)  to run their own FQ server which they could customize.  It was free for players.

Example of customizable data. I still use LORD color codes everywhere. (I’m pretty sure Greg Smith wrote this event btw…):

start_entry
title|MIXED UP VIAGRA MEDICATION IN RETIREMENT HOME
message|"It's like a stabbing ... but different," comments the coroner's office.
give_random|`wThe phone rings - It's Golden Oaks Retirement Home.\n\n`$"We've got some bodies for you..."
mod_bp|40|90
mod_luck|1|5
add_log|`7FUNERAL HOME CHOSEN\n`^Golden Oaks Retirement Home has announced that *NAME* from *HOME* will be hosting services for the recently departed.

Did the grand experiment work?

Well, no.  In retrospect it’s pretty obvious why: why would anyone need to run their own server when calling areas and zip codes no longer exist?  Why would anyone care that they could make a customized version of something so niche as a “funeral parlor simulator”?

The flash login screen. It somehow works perfectly in the latest version of Chrome!  For now…

For some reason I made the server contain a complete HTTP server and run under Windows.  Makes it really easy to setup and use but .. yeah, not  how I’d do it today.

Despite all that, I’m extremely proud of Funeral Quest and have been wanting to repackage a full version with all of the licensing related limitations ripped out for a while. (As I did with Dink Smallwood HD earlier this year)

Well, I finally found the time so here it is.  If anybody actually sets up a working public game of FQ let me know and I’ll help get the word out.  Check the FQ forums if you have problems or questions.

Funeral Quest Server free version (this is all you need to run your own FQ game)

Funeral Quest Flash Source (full flash source for the client part, not required)

The readme file inside:

Funeral Quest
Copyright 2001: Robinson Technologies, all rights reserved

The "should have been released for free eons ago" final free release

Released 3/28 2018
-------------------
This is a special version of the Funeral Quest server that has been modified to no longer need a license, it's the "full version" so to speak.

It can be used to play the game locally or run a real server so hundreds of people can play together.

Notes:

* This version has a few stability fixes since the official last release, I think
* I was nicely surprised that in my local testing (clicking the Logon 1 button) the game seemed to work fine on current versions of both Chrome and Firefox's Flash, if the FQ port remains on the default of 80, anyway. However, who knows what will happen on a real server...
* Run FQServer/fqserver.exe to start the server. Note that when minimized it runs as a tray app, so if it disappeared, check your system tray.

Problems or questions? Check out the official Funeral Quest forums: https://www.rtsoft.com/forums/forumdisplay.php?7-Funeral-Quest

Big thanks to FQ fans and sorry I didn't release this sooner. If you actually get it running and want the full C++ source code in all its MSVC2005 MFC glory, let me know, can probably do that.

-Seth A. Robinson (seth@rtsoft.com)
www.rtsoft.com

Some random screenshots of FQ (some are of the server, and some a browser playing it)

The entire server (which includes its own HTTP server, text to speech notifications, GUI front end) + all Flash client files is less than 1.5 MB zipped

Akiko did all the artwork in Funeral Quest

Many game texts could be customized. A powerful C style scripting language with variable passing and functions was also available. (It’s the same scripting engine that was in Teenage Lawnmower)

It takes skill to read your customers. Training in psychology makes more information about their mood and feelings available.

Maybe now FQ can truly rest in peace.

How to get MySQL Connector/Net working with Unity 2017

So you thought my post on networking a few days ago was boring?  Get ready for database fun!

Normally I would just write about what worked, but in this case I’d rather trek through the foggy world of missteps I went through yesterday to make the various errors searchable, maybe my confusion and pain can be of help to some poor future googler.

Why do you want to directly talk to mySQL from a Unity app?

The first thing you normally see when searching online for help with integrating SQL and Unity is “you idiot, it’s too dangerous to ever connect directly to your server database!”

Well, for the CLIENT APP, that’s correct, it’s a horrible idea because anybody can steal your logon credentials and ruin your database. (I love you Team Meat, but … yeah)

Instead, you need a gateway of some kind (for high scores, this is usually a .php page on a website) that can do things like limit score additions from the same day from the same IP address and other types of ‘hardening’.

But for a game server that needs to access lots of data, a direct connection to a database from your Unity-based dedicated server is absolutely vital.

Step 1: Getting the MySQL Connector/Net files

I downloaded the files for “.NET & Mono” and drag and dropped the 4.5 branch files into my Unity project. (spoiler, this was dumb, I only needed one)

Note: Managed .dll files do NOT have to be dragged into any specific folder like C++ plugins do, anything inside of your Assets  folder is ok, even sub-folders.

Step 2: Activating Unity’s 4.5 .Net  support

From inside of Unity, I clicked on one of the newly added .dll files.  Near the bottom of the property window (it’s easy to miss) it gave this message:

“Plugin targets .NET 4.x and is marked as compatible with Editor, Editor can only use assemblies targeting .NET 3.5 or lower, please unselect Editor as compatible platform.”

To fix this, I changed Unity’s scripting runtime version to ‘Experimental .NET 4.6 Equivalent’.

Step 3: Noticing I added the wrong files

I added ‘using MySql.Data;‘ to a .cs as a test.  It compiled, but when I ran the game I got a “TypeLoadException” without any details.  I installed Unity 2017.3.01f (beta) because I heard it gave better description of these kinds of errors, and it did:

TypeLoadException: Could not load type of field 'MySql.Data.Entity.MySqlMigrationCodeGenerator+<>c:<>9__4_0' (1) due to: Could not load file or assembly 'EntityFramework, Version=5.0.0.0,'

EntityFramework?  Ohhh, so that zip had multiple versions of MySQL Connector and this one wants EntityFramework, whatever that is.  I removed them all (and the .xml files too) and re-added MySql.Data.dll.

Note: You probably don’t need to update to a beta version like I did; this isn’t recommended usually.  Obviously I just like to live dangerously.

Great, no errors and I can do a ‘using MySql.Data;‘ just fine.

Step 4: Adding support for System.Data

I added ‘using System.Data;‘ as we’ll need that too.  Uh oh, got an error there.  So I dragged and dropped  <unity install dir>\Editor\Data\MonoBleedingEdge\lib\mono\4.6-api\System.Data.dll into my project.  (I’m attempting to use the version that comes with Unity that matches the .NET version I’m using)

Well, it now compiles, but now I’m seeing ‘loading script assembly “assets/plugins/system.data.dll’ failed!’ error in the Unity log.  Is this the wrong version?

The error goes away if I use <Unity dir>\Editor\Data\MonoBleedingEdge\lib\mono\2.0-api\System.Data.dll instead. Fine, I’ll use that. (spoiler, this causes problems later)

I wrote some test code using MySQL Connector, it works fine in the Unity Editor!  I’m able to create/destroy/modify SQL databases. No warnings in Unity either!  But uh oh, now I’m seeing this in Visual Studio:

warning MSB3277: Found conflicts between different versions of the same dependent assembly that could not be resolved.".

I guess Visual Studio is set to use a newer version and having the old System.Data.dll referenced is bad?

Not only that, but when I build a Win64 stand-alone version, when running it, I got this error:

NullReferenceException: Object reference not set to an instance of an object
 at MySql.Data.MySqlClient.MySqlPoolManager.GetKey (MySql.Data.MySqlClient.MySqlConnectionStringBuilder settings) [0x00012] in <861c41359f7d4496a8fdf269ae744290>:0 
 at MySql.Data.MySqlClient.MySqlPoolManager.GetPool (MySql.Data.MySqlClient.MySqlConnectionStringBuilder settings) [0x00000] in <861c41359f7d4496a8fdf269ae744290>:0 
 at MySql.Data.MySqlClient.MySqlConnection.Open () [0x0016d] in <861c41359f7d4496a8fdf269ae744290>:0 
 at RTSqlManager.Init (System.String serverIP, System.String userName, System.String userPassword, System.String databaseName) [0x00083] in F:\Unity\SQLTest\Assets\RT\MySql\RTSqlManager.cs:51 
 at SQLTest.Start () [0x00030] in F:\Unity\SQLTest\Assets\_Script\SQLTest.cs:64

Uh… ok, I guess I used the wrong version of the .dll or something, I can see it’s being included but … when I look around, I can see other .DLL’s are coming from directory is actually from <Unity dir>\Editor\Data\MonoBleedingEdge\lib\mono\unityjit and not where I took them from.  Using the System.Data.dll from this “unityjit” dir fixed the Visual Studio warning AND fixed the standalone versions (Win and Linux at least, that’s all I tested).

It works!  So is it fast?

Hallelujah! Everything runs ok, both local in Windows (to test this, you need to setup a MySQL server on Windows) and on linux.  (I’m actually using MariaDB, a MySQL fork, that works fine too)

Speed seems comparable to using the C++ version.  As is my way, I wrote a Unity app with built-in Telnet server to test speeds, here are the results on a dedicated linux server:

Command > ?
/? for help
/create to create testtable
/delete to remove testtable
/add <count> to add a bunch of records
/show <index> to look at a specific record
/stats to show current status of testtable and record count
/seektest <count> to benchmark loading <count> random accounts and reading ID and RandomNum from them
/updatetest <count> to benchmark updating <count> random accounts with a new RandomNum and RandomIndexedNum
/ranktest <count> Rank, benchmark, and view top <count> records


Command > /updatetest 1000
Updating RandomNum and RandomIndexedNum for the first 1000 accounts, each in a different SQL statement
Updated 999 accounts in 5.860773 seconds. (0.005860773 p/account) (time taken: `w5.869883 seconds``)

Command > /seektest 1000
Reading ID and RandomNum for 1000 random accounts in 1000 different SQL SELECT statements...
Read from 1000 accounts in 0.1809998 seconds. (0.0001809998 p/account)
 (time taken: `w0.1875515 seconds``)

Command > /ranktest 1000
Ranking top 1000 random numbers...
#1 ID: 50117 RandomNum: 1999986
#2 ID: 31319 RandomNum: 1999854
#3 ID: 5009 RandomNum: 1999835
#4 ID: 7211 RandomNum: 1999784
#5 ID: 40406 RandomNum: 1999772
#6 ID: 9686 RandomNum: 1999698
#7 ID: 18099 RandomNum: 1999644
#8 ID: 41235 RandomNum: 1999593
#9 ID: 12447 RandomNum: 1999505
#10 ID: 37578 RandomNum: 1999396
(not displaying the rest...)Took 0.02680588 to rank and get 1000 records.
#1 ID: 35 RandomIndexedNum: 19954265
#2 ID: 552 RandomIndexedNum: 19942897
#3 ID: 161 RandomIndexedNum: 19898997
#4 ID: 874 RandomIndexedNum: 19872310
#5 ID: 484 RandomIndexedNum: 19859544
#6 ID: 453 RandomIndexedNum: 19777522
#7 ID: 649 RandomIndexedNum: 19776457
#8 ID: 851 RandomIndexedNum: 19769359
#9 ID: 488 RandomIndexedNum: 19741107
#10 ID: 727 RandomIndexedNum: 19711418
(not displaying the rest...)Took 0.0008430481 to rank and get 1000 indexed records.
 (time taken: `w0.02984619 seconds``)

To break it down (note: this database has 50,000 records):

Update a record: 5 ms

Seek and read from a random record: 0.18 ms

Sort 50k unindexed ints and return the top 1000:  27 ms

Sort 50k indexed ints and return the top 1000:  0.84 ms

Final thoughts

Well, I probably should have created the table with ENGINE=MEMORY as I suspect the update record test was hampered by IO writeback (this system doesn’t have an SSD), but overall I’m happy with these speeds.

I wonder how the speeds would compare with SQLite? Is the fact that I’m using a beta version of Unity change/break anything I did?  I don’t know, but it works fine now so whatever.

Stress testing Unity’s LLAPI, what are the limits?

NetTest – a tool for stress testing the LLAPI, full Unity project download at the bottom!

Seth, what the hell do you know about netcode, anyway?!

Woah, with the attitude already.  Look, I’m all about multiplayer.  Have been for a long time.  I’ve gone from being excited to get two people walking on the same ANSI map in LORD2 in the early 90s to hitting the 70,000 concurrent user mark during busy times in Growtopia.

In addition to writing my own junk, I regularly play with Unity and Unreal and am constantly trying to see how they can fit my developing style – mostly I need them because it’s becoming a losing battle to try to maintain your own 3D engine if you also want to actually design and release games too.

What’s LLAPI, anyway?

LLAPI stands for Low Level Application Programming Interface.  It’s what Unity’s HLAPI (High level) API is built on.  I guess both of those can be called part of UNet. (Unity networking)

Why not just use the HLAPI like a normal person?

That might be ok for some projects, especially round-based networked games without too many players.

When you are targeting MMO level network performance where a server may have to track stats of millions of players and simulate a world for weeks instead of minutes, the first rule is:

Stay as low level as you reasonably can. 

I don’t want someone else’s net code to even think about touching my gameobjects, I just want it to deliver reliable/unreliable packets that I write/read straight bytes to.  That’s it.  I want to handle things like prediction, dead reckoning, and variable serialization, myself!

What’s the deal with UDP, TCP, and WebSockets?

Both UDP and TCP are internet protocols built on top of IP.

TCP is a bidirectional socket based connection that insures your data is correct and sent in the right order.

It’s like calling Jeff, he picks up the phone and you can both talk until one of you puts the phone down.  Oh, and after you say something, you make sure Jeff understood you correctly each time, so that eats up a bit of time.

UDP is is a stateless connection where single packets get sent.  You write your message on a rock and throw it at Jeff’s window.  While it gets there fast, it’s possible you’ll miss and Jeff will never see it. If dozens of people are throwing rocks, he needs to examine each one carefully to figure out which pile to put it with so the text is attributed to the right thrower.  He might read the second rock you threw before the first one.

WebSockets are basically TCP with a special handshake that’s done over HTTP first.  To continue this already questionable analogy, it’s as if you had to call Jeff’s mom for permission, and then she gave you his cell number to call.

In theory WebSocket performance could be similar to TCP but in practice they are much slower and unreliable, I don’t really know why but hey, that’s what we get.  I suspect this will improve later.

For games, UDP is nice because you can send some of your packets fast with no verification that they were received, and others (more important packets, like dying) with verification which sort of makes those packets work like TCP.

That said, most games probably would be ok with a straight TCP  steam as well and having a connected state can make things easier, like easily applying SSL to everything and not worrying about unordered packets.

Why use WebSockets instead of UDP or TCP? Just use UDP or TCP for everything, moran

BECAUSE I CAN’T!  Browsers won’t allow the Unity plugin to run anymore, they just allow straight javascript/WebGL these days.  It hurts.  I mean, I wrote a cool multiplayer space taxi test for Unity using TCP sockets and now nobody can even play it.

What does LLAPI use?

It can read from both sockets (UDP) and WebSockets at the same time and route them so your game can let them play together.  But how fast is it, and does it work?

This brings us to the second rule of netcode:

Use stress tests without game code to notice glaring problems early and more clearly

Which finally brings us to the point of this post.  NetTest is a little utility I wrote that can:

  • Run as a client, server, or both
  • If you connect as “Normal client” you can all chat with eachother
  • Run with a GUI or as headless (meaning no GUI, which is what you want for a server usually)
  • Can open “Stress clients” for testing, a single instance can open 100+ clients sockets
  • When stress mode is turned on, the stress clients will each send one random sentence per second.  The server is set to re-broadcast it to ALL clients connected, so this generates a lot of traffic.  (300 stress clients cause the server to generate 90K+ packets per second for example)
  • Server can give interesting stats about packets, payload vs Unity junk % in the packets, bandwidth and server speed statistics
  • Built in telnet server, any telnet client can log on and get statistics directly or issue commands
  • Tested on Windows, linux, WebGL
  • Supports serializing variables and creating custom packet types, although the stress test only uses strings
  • Everything is setup to push performance.  Things that might mess with the readings like Unity’s packet merging or modifying values based on dropped packets is disabled.  All processing is done as fast as possible, no throttling or consideration to sleep cycles.
  • Keep in mind a real game server is going to also being doing tons of other things, this is doing almost nothing CPU wise except processing packets.  Testing things separately like this make it easier to see issues and know what the baseline is.

ABOUT TEST RESULTS

These tests are presented ‘as is’, do your own tests using my Unity project if you really want exact info and to check the settings.

My windows machine is a i7-5960X, the remote linux box is similar and hosted on a different continent, I get a 200 ping to it.

All packets in NetTest are being sent over the ‘reliable’ channel.  No tests involve p2p networking, only client<>server.

Settings:

config.AcksType = ConnectionAcksType.Acks32; //NOTE: Must match server or you'll get a CRC error on connection
config.PingTimeout = 4000; //NOTE: Must match server or you'll get a CRC error on connection
config.DisconnectTimeout = 20000; //NOTE: Must match server or you'll get a CRC error on connection
config.MaxSentMessageQueueSize = 30000; 
config.MaxCombinedReliableMessageCount = 1;
config.MaxCombinedReliableMessageSize = 100; //in bytes
config.MinUpdateTimeout = 1;
config.OverflowDropThreshold = 100;
config.NetworkDropThreshold = 100;
config.SendDelay = 0;

TEST 1: 300 CLIENT IDLE (LOCALHOST, WINDOWS BOX)

Ok, here I’m curious how many bytes LLAPI is going to use with clients doing nothing.

Server is set to localhost.  I’ve got 4 NetTest’s running – the one on the bottom right has had “Start local host” clicked, and the three others each have added 100 stress clients.

Results:

  • Around 30 bytes per stay alive packet per client.  (23KB total over 7.74 seconds, sent 2.2 packets per client over 8 seconds, so roughly what I would expect for the 4000 ms ping timeout setting)  Are stay alives sent over the reliable channel?  Hmm.
  • Adding and removing test clients is a blocking operation and gets progressibly slower as more are added – why does it get slower and cause my whole system to act a bit weird?
  • While closing the instances looks normal, the instances are actually spending an additional 30 seconds or so to close sockets that were opened
  • Side note: Having a client ping the host (at localhost) takes about 2 ms.  (normally this would be bad, but given that it can’t check until the next frame, then once more to get the answer, this seems to match the framerate decent enough)

I’m not going to worry about the socket slowness, this may be related to a windows 10 resource thing.  I’ll launch with less clients in future tests to help though.  The server itself has zero speed/blocking issues with actually connecting/disconnecting though, so that’s good.

TEST 2: 200 CLIENT STRESS (LOCALHOST SERVER, WINDOWS BOX)

Ok, now we’re getting serious.  I have 5 instances – 4 have 50 stress clients, 1 is the server.  (I could have run the server on one of the stress client instances, but meh)

First I enabled 50 clients in “stress mode” – this should generate 10,000 packets (50*200) per second.  The server gets 50 lines of text from clients, then broadcasts each one to each of the 200 clients.  Huh, stats show it only sending around 6,000 packets per second, not 10,000.  However, we can see all the lines of text are being delivered.

That seemed fine so I upped it to all 200 clients doing the stress test – this should cause the server to send 40,000 packets a second (and receive 200 from the clients).

The stats show only 24,549 packets and 5.3 MB per second being sent. About 15% is unity packet overhead, seems like a lot but ok.  Server FPS is good, slowest frame was 0.02 (20 ms) so not bad.

Is unity combining packets even though I set config.MaxCombinedReliableMessageCount = 1; ?  Oops, I bet that needs to be 0 to do no combining.

I also notice 4,346 per second packets being received by the server.  200 for my messages, and I assume the rest are part of the “reliable guarantee” protocol they are doing for UDP where the clients need to confirm they received things ok.  Oh, and UNet’s keep alives too.

In the screenshot you can see an error appearing of “An abnormal situation has occurred: the PlayerLoop internal function has been called recursively… contact customer support”.  Uhh… I don’t know what that’s about.  Did I screw up something with threads?  It didn’t happen on the server, but one of the four client instances.

I let it run at this rate for a while – a few clients were dropped by the server for no reason I could see.   Not entirely stable at these numbers I guess, but possible it would be if I ran it on multiple computers, it’s a lot of data and ports.

TEST 2: 100 CLIENTS STRESS (REMOTE SERVER, HEADLESS LINUX)

As nice as it is to be able to turn on both the server (localhost) and the client directly from the same app in the Unity editor to test stuff, eventually you’ll probably want to get serious and host the server on linux in a data center somewhere.  This test let’s us do that.

I’ve changed the destination address to the remote IP. (You can click the GameLogic object to edit them, I was too lazy to make a real GUI way))

I have a .bat file that builds the Unity project (on Windows) and copies it to the remote server, then triggers a restart.  It’s run on Linux like this:

cd ~/NetTest
chmod +x ./linux64headless
./linux64headless -batchmode -nographics -logFile log.txt

Here is a screenshot of the 100 clients (using one instance of NetTest) connecting to the linux server (which is also NetTest) (And no, that’s not the real port/ip, stop trying to hack me, Mitnick)

So let’s try to digest this data.

  • NetTest Client: The 100 clients are self reporting receiving about 1,150,000 (1.15 MB) bytes of actual payload data per second. (10000 lines of text).  Zero packets in the send queue
  • Windows system: Windows says 12.1 Mbps per second, so 1.5 MB, close enough, who knows what else I’m streaming/downloading
  • Linux system: iotop shows server sending 10.Mb (1.25 MB per second) so that seems right
  • Linux NetTest: (using telnet) it’s reporting it’s sending 1.22 MB per second (only 3,458 packets per second, yeah, definitely some packet merging happening)  The total stats went negative, I ran this a while, uh… I guess I need to be using larger numbers, probably rolled over.
  • Linux: Server is running at 1100 FPS, so zero problems with speed.  You’re probably thinking “Hey, it’s headless, what do you mean frames per second? What frames?”, ok, fine, it’s really measuring “update()’s per second”, but you know what I mean.  Slowest frame out of the 90 second sample time was 0.0137 (13 ms)
  • I don’t have it in the screenshot, but top shows the server is not too power hungry, its process seems to use 3% (no load) to 15% (spamming tons of packets).  Keep in mind we aren’t actually DOING anything game related yet, but for just the networking, that isn’t bad

I tried 200X200 (4x the total bandwidth & packets, so 5 MB a second) but the outgoing packet queue started to fill up and I started dropping connections.

TEST 3: WebGL (REMOTE SERVER, HEADLESS LINUX)

I’m not going to bother making pics and somehow this post already got way out of control, but here is what seemed to be the case when I played with this earlier:

  • No WebSocket stats are included in the stuff UNet reports – “the function called has not been supported for web sockets communication” when using things like NetworkTransport.GetOutgoingFullBytesCountForHost on it
  • WebGL/WebSockets be slow.  Real slow. If pushed the bandwidth to over 18KB per second (per Chrome tab), packets started to get backed up.  I don’t know if things are getting throttled by the browser or what.  The server eventually freaks out when the send queue gets too big and starts throwing resource errors.
  • When creating stress test clients, a single WebGL tab seems limited to 14 WebSockets max (Chrome at least)
  • I suspect there could be issues with a naughty WebSocket filling up the queue and not sending acknowledges of receives, your game probably needs to carefully monitor packets sent/received to disconnect people quick if they abuse the connection
  • WebSocket queue/etc issues don’t affect or slow down normal connections, so that’s something
  • Mixing WebSocket and normal Sockets (UDP) clients work fine, that’s a very handy thing to be able to do

Final thoughts

All in all, I think directly using LLAPI in Unity 2017.1+ is promising.  Previously, I assumed I’d need to write my own C++ server to get decent socket performance (that’s how I did the Unity multiplayer space taxi demo) but now I don’t think so, if you’re careful about C#’s memory stuff.

Issues/comments:

  • How is it ok that Unity’s text widget/scroller can only show like 100 lines of text? (65,535 vert buffer errors if you have too much text, even though most of it is off screen)  Maybe I’m doing something wrong
  • Unity’s linux builds don’t seem to cleanly shutdown if you do a pkill/sigterm. OnApplicationQuit() does not get run, this is very bad for linux servers that could restart processes for reboots or upgrades. (if it’s a dedicated server you can work around it, but still!)
  • WebGL builds can’t do GetHostAddresses() so I had to type the real IP in for it to connect (I should try switching to the experimental 4.6 net libs)
  • To my great surprise, my telnet host contined to run and send answers even after stopping the game in the editor.  How could that game object still exist?! Threads and references are confusing, I fixed it by properly closing the telnet host threads on exit.
  • The Unity linux headless build worked perfectly on my CentOS 7.4 dedicated server, didn’t have to change a single thing
  • It’s weird that you’ll get CRC errors if PingTimeout/AcksType/DisconnectTimeout don’t perfectly match on the client and server. I wonder why there is not option to just let the server set it on the client during the initial handshake as you want to be able to tweak those things easily server-side.  I guess the client could grab that data via HTTP or something before the real connect, but meh.

NetTest download

Download the NetTest Unity project with full source here

Note:  I left in my .bat files (don’t mock the lowly batch file!), thought they may be useful to somebody.  They allow you to build everything for linux/win/webl, copy to the remote servers and restart the remote server with a single click.  Nothing will work without tweaking (they assume you have ssh, rsync etc working from a dos prompt) but you could probably figure it out.

Source is released under the “do whatever, but be cool and give me an attribution if you use it in something big” license.