Door game source code release – RTReader

Many moons ago I made a door game scripting engine that read “ref files” (short for reference, I think) to control  what the users see.  Later, this reference file system was used as the base for LORD 2: New World.

Someone asked me today for the source and as I don’t think there are any legal entanglements with releasing this, here ya go.  (About the LORD/TEOS/LORD2 source, I’ve been trying to buy back the rights forever so I can legally release them too)

RTReader V??? source code (probably incomplete and can not actually be compiled, written in Turbo Pascal 6, requires Scott M. Baker’s door driver)

Note:  Does anybody perchance have the RTReader door game .zip file I distributed back in the day?  I think it was a weird kind of “RTsoft company update and virtual interview with Seth” thing, please drop a comment if you know where to find it, thought it might be good for a laugh. My google-fu failed.

UpdateHere is the RTReader thing I was talking about (Thanks Jay).  I’ve packaged it with DosBox, just click play.bat and it will run.

It’s just as hilarious and bad as I expected, complete with tons of misspellings and random things such as a virtual interview and a “game” that ends with a madlibs style story.   It calls itself “Bloody Claw Newsletter Issue 2 from June, 1995”.  So there must be an “issue 1” out there somewhere?  Did an issue 3 ever make it out the door? Hrm.

Input lag fun – measuring Atari 2600 latency from controller to display with an Arduino

Input lag.  It’s a catch-all name people use when talking about the latency that gets added in the various places between when you push a button and finally see results on your screen.

Many (especially console) games these days are designed to hide it because the developers cannot predict the latency of the display device the player is using.

There are often additional considerations such as video drivers, vsync, “Game mode” display options, refresh rate, back buffering and dealing with high latency wireless controllers.

If I add 100 milliseconds of additional lag to Red Dead Redemption 2, I’d doubt you’d notice with its mushy-ass controls.  But if you try that with old school platformers and bullet hells designed for ultra-low lag, well, it ain’t gonna be pretty.

If you’re skeptical of the difference input lag can make, try this:

Super Mario Bros. 2 (Japan) is just mean. See that mushroom? It kills you!

Play the murderous Super Mario Brothers 2 (Japan) directly from a Famicom on an old school CRT and then try the same thing on your Raspberry Pi, PC, or even the same NES console through an upscaler/LCD tv.

As you switch back and forth you’ll probably feel the difference.  The game is more difficult and less comfortable with the extra lag.

It’s not just about reaction times – there is this thing your brain does where it’s forced to jump and move slightly earlier than the onscreen action.  We can all automatically do it, sure, but it’s… different.  It doesn’t feel as connected.  It’s too.. I don’t know, milky.  A lot depends on the number of frames missed as well as when the console polls. (for example, the Atari 2600 polls 30 times a second, during the vertical blank interrupt but )

This goes for much of the 8 and 16 bit action content from consoles and computers of yesteryear.

So real hardware and CRTs are the only way to go for fast response controls?

Woah, settle down, I didn’t say that!  Retro emulation is astonishing and with the right gear and settings it should be possible to match or even beat old-school latency in specific cases such as with Run Ahead on the NES.

That said, I have not been able to do it yet with my setups.  Additionally, there are sometimes trade-offs like screen tearing and visual artifacts when you’re aiming at ultra-low latency.  I’m sure things will continue to improve given the recent focus on reducing input lag on both displays and controllers.

(Note: I was going to say “twitch gaming”  instead of “fast response controls” but I’m guessing that term is too ambiguous these days)

Measuring input lag accurately

Instead of getting lost in subjective testing by “feel”, let’s get scientific about it.  For my purposes, I wanted to measure the exact latency between a button press on an Atari 2600 console and the display pixels changing.  The full trip.

<disclaimer – I’m an electronics amateur and don’t know what I’m doing.  Breaking stuff is how you learn, right?> 

I used a cheap Arduino Uno with an LCD/button shield (they call things you stick on them shields, ok?) to do it.  It’s so simple you don’t even need a breadboard!

The light sensor

First I wired up an LDR (Light Dependent Resistor) to the board’s analog in pin A0, and connected the other end to the ground.  We can now detect light with a analogRead(A0) command.  Oh right, I also put a 100K resistor in there.

To get the light sensor to physically stick on the screen (so it will hover over the pixels in question) I commandeered the suction cup that came in a iFixit kit and glue-gunned a rubber earbud cover to house it.

Just nod and smile please, without commenting how there is glue everywhere and it looks ridiculous.

My light sensor returns a number between 300 and 1024 or so.  When stuck onto a monitor more like 800 to 1000 (black and white… black pixels still have a lot of light coming through I guess) but good enough.

Getting the Arduino Uno to push the Atari fire button digitally

Next I cut the cable off a broken Atari 2600 controller (it was a cheap clone from Ebay that I broken during a particularly exuberant C64 performance) and using this diagram figured out the wires for the fire button and ground.

I connected the controller ground wire to the Arduino’s ground pin, then the fire button wire in the D2 pin.  I can now control the fire button like this:

 pinMode(C_ATARI_BUTTON_PIN, OUTPUT);

//To turn button off
 digitalWrite(C_ATARI_BUTTON_PIN, HIGH);

//To turn the button on
 digitalWrite(C_ATARI_BUTTON_PIN, LOW);

I didn’t know you could directly wire it like that, mixing the 5v high signals from both the Atari and Arduino, but whatever, it works.  Read this post by BigO to understand why setting it to LOW causes it to be “on” as far as the Atari is concerned.

I noticed if the Arduino is NOT powered buts its joystick lead is plugged into the Atari, there are weird video glitches.  I guess due to the unpowered Arduino draining amperage from the fire button lead, enough to cause fluctuations in the entire system?  Ignore any smoke, move along.

Adding support for more buttons would be as easy as plugging the additional wires into more Arduino pins.  In the picture of the whole device above, it’s only the Red and Yellow wires I’m using, the blue/white ones aren’t connected to anything.

The code

All that’s left is to write some code so the device can be controlled with the LCD shield’s buttons.   Here’s what those buttons do:

  • Select – Show current light level reading
  • Left – Nothing
  • Up – Increase light change timing required to trigger
  • Down – decrease light change time required to trigger
  • Right – Start measuring.  (Will cause the Atari’s fire button to click, then it waits until the light has changed enough.

Here is the code for the Arduino sketch. (I did all this in a messy few hours, don’t judge me)

Tip: I leave the button in the “pressed” state for 100 MS (I guess 33.333 MS would technically be enough for an Atari 2600, but whatever, doesn’t matter), and I look for about a 60 change from the light sensor to count as “hey, this part of the screen has  definitely changed, stop the timer, we’re there”.

You can’t see it, but I’m pulling the Reset button on the Atari 2600 between tests so it’s on the right screen for the Arduino to send the “fire button” when I start a measurement.  The game is Xenophobe.

Testing results:  Atari 2600 console on CRT

  • Sensor in the upper left of the CRT return between 0 and 33 MS.
  • Sensor in the buttom left of the CRT returns between 16 and 33 MS.

This seems about what it should be, give or take 1 MS or so?  It’s possible to get near 0 MS from button push to pixel display.  I mean, I guess I knew it was, but it’s still fun to measure it.

So why did I use the Xenophobe cartridge?  Because it’s just the first game I found that clearly changes a dark block of the screen to a lighter color instantly when a button is pressed. (I wouldn’t want to use light to dark due to possible ghosting issues)

There are probably homebrew roms built to do this for most system but I couldn’t find one with a cursory googling so here we go.

Testing results: Atari 2600 console with upscaler, various video switchers & old Dell LCD monitor

  • Sensor in upper left of panel returns between 79 and 130 MS

Ouch.  Well, I knew it wasn’t going to be good, I can only imagine how bad it would be with a Pi or something instead.  Anyway, I won’t go serious into testing (I’m no My life In Gaming)  or my exact hardware in my retro area (it’s weird…), I just want to be ready for when I need to compare latency on my setups down the road.

Conclusion & random thoughts

I’d like to test the full “controller to display” latency on my Raspberry Pi & main computer setups as well but I think that means I’d have to hack into a 360 controller so the Arduino can control the button as we did with the Atari.   Maybe later.

Would be wondrous to get to a place where we could once again write games counting on the C2S  (controller to screen) lag being low and knowing everybody is getting the best experience.

You always want to know your server ping time when you play online, right?

Well, maybe we could start building in latency test systems so a console (using its camera? or maybe all TV/monitors’s should have a tiny light sensor in a corner that could be queried via HDMI) would know its own latency (both visual and auditory) and a game could use that data to automatically sync up stuff for rhythm games, adjust game timing of quick-time events, or even to just display a “congratulations, you’ve got horrible latency” warning if needed.

Here’s a Chrome extension that scrapes Sony’s website to get a list of owned Playstation games

If you’ve ever wanted a list of your digitally purchased/downloaded Playstation games (I have over 500 <sigh>) I wrote this chrome extension that can do it as a test for something else.

It’s crap because it will break if the Sony site changes at all, as well as missing games that weren’t purchased digitally (disc games, for example) but hey, here it is.

Click here for Github source & install/usage instructions

The data it creates is in json format and looks like this:

"games": [
    {
      "title": "Judgment",
      "Size": "30.75GB",
      "PurchaseDate": "7/4/2019",
      "Platforms": [
        "PS4"
      ],
      "productID": "UP0177-CUSA13186_00-JUDGMENTRYUGAENG"
    },
    {
      "title": "Borderlands: The Handsome Collection",
      "Size": "28.64GB",
      "PurchaseDate": "7/4/2019",
      "Platforms": [
        "PS4"
      ],
      "productID": "UP1001-CUSA01401_00-BORDERLANDSHDCOL"
    }, ... and so on

If I cross referenced it with trophy data it would be more accurate.  If anyone knows a better way to get at this data (one that doesn’t break if the user isn’t using English for example…) please let me know.

No plans to add anything else to this but  wanted to throw up a post about it so anybody else working on something similar could find the source if needed.

How to monitor your kids’ online experience on the Nintendo Switch (like Fortnite voice chat)

The problem

“Oppia,  Oppia,  Oppai!”   My younger kid was chanting this word over and over into his headset mic while playing Fortnite on the Switch.

Why is he saying this? I tell him to flip the monitor speaker on.  Some laughing dumbasses were telling him if he said the word (which means “breasts” in Japanese) enough times, they would buy him a skin.  (Fortnite skins are giftable right now… this presents griefers with a lot of ammo to screw with naive children)

Note: When I say “monitor speaker” I just mean a tiny speaker to monitor the conversation, nothing to do with a computer monitor or TV.

How I deal with this (and worse)

Instead of freaking out and forbidding voice chat completely, I prefer to use it as a teachable moment to prepare him for his inevitable technological future.

So how do you know when someone is lying to you?  How do you know what’s true and false?

If I can teach him to be skeptical, maybe he won’t get suckered later in life.  <Looks out window, sees homeopathy, magical wallets, magnetic sports bracelets… sigh>

What behavior crosses the line?  How does muting and banning work? Friend only chat?  What about strangers in a friend’s squad?

In the all too near future he’ll be flying solo, but for now I can ride copilot when issues come up and monitor the conversation with the right cables and a tiny speaker.  Works better than my old way: grabbing the headphones and interrupting the game.

Want this?  Here’s what to buy

Here’s what I used:

Note: I linked to the items on amazon/ebay as examples, this isn’t an endorsement of any particular seller or product, just to show what the correct cables look like… until the links break, anyway.

3.5mm TRRS Male to Dual TRRS Female Stereo 4-Pole Splitter Cable  (Amazon) <– This is the trickiest part to buy because you can’t really see from a picture if the female ends are wired correctly (you want a full TRRS jack inside and not a splitter to a TRS or something)

3.5mm TRRS to 2 TRS Audio Headset Mic Y Splitter Cable Adapter (Ebay)

Mini 3.5mm Hamburger Speaker USB Rechargeable (Ebay)

More details with crappy diagram photoshop

A normal headphone cable (TRS) carries three conductors, left channel, right channel, and an audio ground.  A TRRS cable adds one more wire so it can hold four – the extra conductor is used for the mic.  (The mic shares the same audio ground)

If you directly plugged in a TRS cable into a TRRS jack, you’d accidently short the mic and ground together which would disable the mic input.  That’s why we need the TRRS to headphone/mic splitter, so we can just plug the speaker into the headphone part.

The mic end of the splitter is unused.  See the pic at the top of this post to see how to plug stuff in, it’s pretty simple.

If the mic is picking up sounds from the speaker, move it farther away or turn the speaker volume down.

Having written a system-wide live content scanner for an MMO I can tell you that there is no way you could overestimate how shitty (some) people can be online, so please keep an eye on your kids and don’t trust any system to do it for you.

Conclusion

Is there any better way to do this?  Maybe there is a tiny speaker out there that already has a TRRS passthrough built in? That’d be cleaner.

If using a dock, the TV speaker might be an option but the Switch has no option to “Play audio via headphones and HDMI simultaneously”, so it always mutes HDMI when headphones are plugged in.

Despite the risks and warnings above, I’m ok with the current reality of online gaming as an important school-wide social activity.  Earning “screen time” provides motivation for my son to finish his homework and hey, at least he’s getting exercise.

Universal Game Translator – Using Google’s Cloud Vision API to live-translate Japanese games played on original consoles (try it yourself!)

Why I wanted a “translate anything on the screen” button

I’m a retro gaming nut.  I love consuming books, blogs, and podcasts about gaming history.  The cherry on top is being able to experience the identical game, bit for bit, on original hardware.  It’s like time traveling to the 80s.

Living in Japan means it’s quite hard to get my hands on certain things (good luck finding a local Speccy or Apple IIe for sale) but easy and cheap to score retro Japanese games.

Yahoo Auction is kind of the ebay of Japan.  There are great deals around if you know how to search for ’em.  I get a kick out of going through old random games, I have boxes and boxes of them.  It’s a horrible hobby for someone living in a tiny apartment.

Example haul – I got everything in this picture for $25 US! Well, plus another $11 for shipping.

There is one obvious problem, however

It’s all in Japanese.  Despite living here over fifteen years, my Japanese reading skills are not great. (don’t judge me!) I messed around with using Google Translate on my phone to help out, but that’s annoying and slow to try to use for games.

Why isn’t there a Google Translate for the PC?!

I tried a couple utilities out there that might have worked for at least emulator content on the desktop, but they all had problems.  Font issues, weak OCR, and nothing built to work on an agnostic HDMI signal so I could do live translation while playing on real game consoles.

So I wrote something to do the job called UGT (Universal Game Translator) – you can download it near the bottom of this post if you want to try it.

Here’s what it does:

  • Snaps a picture from the HDMI signal, sends it to google to be analyzed for text in any language
  • Studies the layout and decides which text is dialog and which bits should be translated “line by line”
  • Overlays the frozen frame and translations over the gameplay HDMI signal
  • Allows copy/pasting the original language or looking up a kanji by clicking on it
  • Can translate any language to any language without needing any local data as Google is doing all the work, can handle rendering Japanese, Chinese, Korean, etc  (The font I used is this one)
  • Controlled by hotkeys (desktop mode) or a control pad (capture mode, this is where I’m playing on a real console but have a second PC controller to control the translation stuff)

In the video above, you’ll notice some translated text is white and some is green.  The green text means it is being treated as “dialog” using its weighting system to decide what is/isn’t dialog.

If a section isn’t determined to be dialog, “Line by line” is used.  For example, options on a menu shouldn’t be translated all together (Run Attack Use Item), but little pieces separately like “Run”, “Attack”, “Use item” and overlaid exactly over the original positions.  If translated as dialog, it would look and read very badly.

Here are how my physical cables/boxes are setup for “camera mode”. (Not required, desktop mode doesn’t need any of this, but I’ll talk about that later)

Happy with how merging two video signals worked with a Roland V-02HD on the PlayStep project, I used a similar method here too.  I’m doing luma keying instead of chroma as I can’t really avoid green here. I modify the captured image slightly so the luma is high enough to not be transparent in the overlay. (of course the non-modified version is sent to Google)

This setup uses the windows camera interface to pull HDMI video (using Escapi by Jari Kompa) to create screenshots that it sends to Google.  I’m using an Elgato Cam Link for the HDMI input.

Anyway, for 99.99999999% of people this is setup is overkill as they are probably just using an emulator on the same computer so I threw in a “desktop mode” that just lets you use hotkeys (default is  Ctrl-F12) to translate the active Window. It’s just like having Google Translate on your PC.

Here’s desktop mode in action, translating a JRPG being played on a PC Engine/TurboGrafx 16 via emulation. It shows how you can copy/paste the recognized text if want as well, useful for kanji study, or getting text read to you.  You can click a kanji in the game to look it up as well.  (Update: It now internally can handle getting text read as of V0.60, just click on the text.  Shift-Click to alternate between the src/dest language)

Try it yourself

Before you download:

  • All machine translation is HORRIBLE – this is no way replaces the work of real translators, it’s just (slightly) better than nothing and can stop you from choosing “erase all data” instead of “continue game” or whatever
  • You need to rename config_template.txt to config.txt and edit it
  • Specifically, you need to enter your Google Vision API key.  This is a hassle but it’s how Google stops people from abusing their service
  • Also, you’ll need to enable the Translation API
  • Google charges money for using their services after you hit a certain limit. I’ve never actually had to pay anything, but be careful.
  • This is not polished software and should be considered experimental meant for computer savvy users
  • Privacy warning: Every time you translate you’re sending the image to google to analyze.  This also could mean a lot of bandwidth is used, depending how many times you click the translate button.  Ctrl-12 sends the active window only, Ctrl-11 translates your entire desktop.
  • I got bad results with older consoles (NES, Sega Master System, SNES, Genesis), especially games that are only hiragana and no kanji. PC Engine, Saturn, Dreamcast, Neo-Geo, Playstation, etc worked better as they have sharper fonts with full kanji usually.
  • Some game fonts work better than others
  • The config.txt has a lot of options, each one is documented inside that file
  • I’m hopeful that the OCR and translations will improve on Google’s end over time, the nice thing about this setup is the app doesn’t need to be updated to take advantage of those improvements or even additional languages that are later supported

After a translation is being displayed, you can hit ? to show additional options.  Also, this is outdated, use the real app to see the latest.

5/8/2019 – V0.50 Beta – first public release, experimental
5/13/2019 – V0.51 Beta – Added S to screenshot, better error checking/reporting if translation API isn’t enabled for the Google API key, minor changes that should offer improved translations
5/30/2019 – V0.53 Beta – Added input_camera_device_id setting to config.txt for systems with multiple cameras.  Moves mouse offscreen for “camera” mode captures
9/5/2019 – V0.54 Beta – Fixes crash on startup problem some people had, adds “audio|none” config.txt command to optionally disable all sound.  Added “minimum_brightness_for_lumakey” setting to config.txt in case the default isn’t right
9/15/2019 – V0.60 Beta – New feature, text to speech!  You’ll need to enable Google’s Text To Speech API, Fixed a crash bug, added some in-app persistent settings, gamepad can now move around the cursor and click things.  Controls changed a bit. Added automatic reading of detected dialog, can choose to read src or dest langs, can hide text overlays if you want now.  A few new options in the config.txt. Switched to FMOD audio, SDL_Mixer has buggy mp3 playback which was causing some me grief. Changed the translate button sound to something more soothing.

Note: I plan to open source this, just need to get around to putting it on Git, if you’re someone who would actually do something with the source, please hassle me into doing it.

Download Universal Game Translator for Windows (64-bit) (Binary code signed by Robinson Technologies)

Conclusion and the future

Some possible upgrades:

  • Built-in text to speech on the original dialog (well, by built in I mean using Google’s text to speech API and playing it in UGT, easier than the copy and paste method possible now)
  • A built in Kanji lookup also might be nice,  Jim Breen’s dictionary data could work for this.
  • My first tests used Tesseract to do the OCR locally, but without additional dataset training it appeared to not work so hot out of the box compared to results from Google’s Cloud Vision.  (They use a modified Tesseract?  Not sure)  It might be a nice option for those who want to cut down on bandwidth usage or reliance on Google.  Although the translations themselves would still be an issue…

I like the idea of old untranslated games being playable in any language, in fact, I went looking for famous non-Japanese games that have never had an English translation and really had a hard time finding any, especially on console.  If anyone knows of any I could test with, please let me know.

Also, even though my needs focus on Japanese->English, keep in mind this also works to translate English (or 36 other languages that Google supports OCR with) to over 100 target languages.

Test showing English being translated to many other languages in an awesome game called Growtopia