Using computer vision to enforce sleeping pose with the Jetson Nano and OpenCV

(special thanks to Eon-kun for helping demonstrate what it looks like)

Imagine you HAVE to sleep on your back for some reason and possibly restrict neck movement during the night. Here are some options:

  • Tennis balls strapped to sides
  • Placing an iphone on chest/pocket and using an app (SomnoPose) that monitors position with the accelerometer and beeps when it detects angle changes. (it works ok but the app is old and has some quirks like not running in the background)

The above methods are missing something though – they don’t detect head rotation. If you look at the wall instead of the ceiling while not moving your body, they don’t know.

The tiny $99 Jetson Nano computer combined with a low light USB camera can solve this problem in under 100 lines of Python code! (A Raspberry Pi would work too)

The open source software OpenCV is used to processed the camera images. When the program can’t detect a face, it plays an annoying sound until it does, forcing you to wake up and move back into the correct position so you can enjoy sweet silence.

If you’re interested in playing with stuff like this, I recommend Paul McWhorter’s “AI on the Jetson Nano” tutorial series, the code below can be used with that.

I’m really excited about the potential of DIY electronics projects like this to help with real life solutions.

The Pi and Nano have GPIO pins so instead of playing a sound, we could just as easily activate a motor, turn a power switch on, whatever.

Of course instead of just tracking faces, it’s also possible to look for eye, colors, shapes or cars, anything really.

The Python code listing for Forcing you to sleep on your back

import cv2
import time
from playsound import playsound
import os

dispW=1024
dispH=768
timeAllowedWithNoFaceDetectBeforeWarning = 22
timeBetweenWarningsSeconds = 10

timeOfLastFaceDetect = time.time()
timeSinceLastDetect = time.time()
timeOfLastWarning = time.time()
warningCount = 0

def PlayWarningIfNeeded():
    global timeBetweenWarningsSeconds
    global timeOfLastWarning
    global warningCount

    if time.time() - timeOfLastWarning > timeBetweenWarningsSeconds:
        print ("WARNING!")
        warningCount = warningCount + 1
        os.system("gst-launch-1.0 -v filesrc location=/home/nano/win.wav ! wavparse ! audioconvert ! audioresample ! pulsesink")
        timeOfLastWarning = time.time()


bCloseProgram = False

cv2.namedWindow('nanoCam')
cv2.moveWindow('nanoCam', 0,0)
cam = cv2.VideoCapture("/dev/video0")

cam.set(cv2.CAP_PROP_FRAME_WIDTH,int(dispW))
cam.set(cv2.CAP_PROP_FRAME_HEIGHT,int(dispH))
cam.set(cv2.CAP_PROP_FPS, int(10))
face_cascade = cv2.CascadeClassifier('/home/nano/Desktop/Python/haarcascades/haarcascade_frontalface_default.xml')
fnt = cv2.FONT_HERSHEY_DUPLEX

while True:

    ret, frame = cam.read()
    frame = cv2.flip(frame, 0) #vertical flip

    #rotate 90 degrees
    #frame = cv2.rotate(frame, cv2.ROTATE_90_CLOCKWISE)
    gray=cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    faces = face_cascade.detectMultiScale(gray, 1.3, 5)

    for (x,y,w,h) in faces:
           cv2.rectangle(frame, (x,y), (x+w, y+h), (0,255,0), 4)
           timeOfLastFaceDetect = time.time()
   
    timeSinceLastDetect = time.time()-timeOfLastFaceDetect
   
    if timeSinceLastDetect > timeAllowedWithNoFaceDetectBeforeWarning:
         PlayWarningIfNeeded()
        
    text = "Seconds since face: {:.1f} ".format(timeSinceLastDetect)
    frame = cv2.putText(frame, text, (10,dispH-65),fnt, 1.5,(0,0,255), 2)

    text = "Warnings: {} ".format(warningCount)
    frame = cv2.putText(frame, text, (10,dispH-120),fnt, 1.5,(0,255,255), 2)

    cv2.imshow('nanoCam',frame)
    if cv2.waitKey(10)==ord('q') or cv2.getWindowProperty('nanoCam',1) < 1:
        bCloseProgram = True
    
    if (bCloseProgram):
        break

cam.release()
cv2.destroyAllWindows()

Seth answers Growtopia questions (FAQ)

What is a F.A.Q.?

It means frequently asked questions.  It’s a single webpage that I can direct people to for detailed answers to common questions.  I also have one for my BBS games here which I sold to another company.

I’ll update it to modify and add questions as needed. (last updated 8/17/2020)

Why did you sell Growtopia in 2017?


I can only speak for myself for all of these answers, but after four years Growtopia had gotten too big. Too many players, not something I’d ever consider a problem before it happened.

While we were able to hire great customer support folks, it was still only Hamumu and myself developing and running the game.  

We were stretched too thin.  When you’re taking money for a product, you have a responsibility. 

The legal responsibility grows proportionally to your player count and how much they’re spending.  Too much cash was being spent for us to claim ignorance if something went wrong.  

What if a backup failed?  What if we screwed up on COPPA compliance?  The excuse of “hey, we’re just two people” wouldn’t cut it anymore.

The pressure builds.  After four years of running Growtopia I started to have regular nightmares, usually involving backups failing during a rollback or other server problems.

In the end, selling the game meant it could continue to see new features and grow, without us having to become bosses and have employees and all that, things we never wanted.

We got to use the profit from the sale to remain small and happy indies that can work on what we want, when we want.

In my experience that’s extremely rare you get a chance like that so I feel really lucky it worked out.


Would you ever come and work on the game as a real dev?

No. Assuming I was ever asked, I couldn’t accept the job.

Working on an MMO is not something you can do part time, you have to eat and breath it and get woken up by alarms at 3 AM when something breaks.

My life has changed now, less stress and very different kinds of challenges including running a cafe and hotel with Akiko. (Hotel? Well, er, it’s only one (amazing) room, but we are officially licensed as both a restaurant and hotel)

Growtopia is being ruined!

Ok, that’s not really a question but it’s something people say to me sometimes, especially when the server is down.

People tend to remember the positives of history rather than the negatives.  But if you go back to the forums there was no time after the first year we ran the game there were not “Growtopia is ruined and dying now” threads being made.

The life of this kind of game is all about change – breaking, fixing, adding, and very little removing.

The bigger the change or addition, the greater the risk.  The more systems piled on top of each other it is increasingly challenging it is to keep it all running smoothly. I have a lot of sympathy for the developers as I know their pain. :)

I hope they succeed and the game runs forever but there isn’t really any kind of comment I can make on this kind of thing, especially since I don’t play much or know what the issues are.

If the game isn’t for you, try something else for a while, it’s perfectly normal and ok to get tired of a game and bounce around.


I was banned for no reason, please help me by contacting Ubisoft

Sorry I can’t.  I don’t have access to the logs or support system, I’m the wrong person to ask. 

It would be really awkward if I was trying to do any kind of modding while not actually on the support team!

You need to go through the normal Growtopia help channel here.

Do you still play GT?  Can you sign my board?  Take a selfie with me in the game?

This is kind of complicated to answer.  Currently my @Seth account remains completely locked (even from me) unless I specifically ask for it to be unlocked for a period.  I like it this way.

The reasons for this are: security (it’s VERY bad if this account gets hacked), performance (if people see I’m online, they ALL msg me at once, can cause problems), and also something I call “the rumor wildfire” – sometimes a rumor starts that a hacker is using my account and mods and support gets flooded with emails about this emergency – so it’s important that they know in advance when I’ll be online.


So in the end, I almost never ask that my account be unlocked because it’s a hassle for everybody and I don’t want to cause any trouble.  (Ubisoft has been very cool about it, but I still feel bad)

I tried using another account but.. Mehhh, it’s just not the same.

It’s pretty rare that I login these days for anything other than P.A.W. for reasons stated above.  

However, if I’m online in the game I will make every effort to accommodate those kinds of requests, I use /rgo a lot.


Note: One of the conditions of the sale was nobody would ever pretend to be us, so if you ever see @Seth or @Hamumu talking to you, it’s really us!

Are you annoyed that every tweet or video you make has Growtopia related comments and questions on it?


The truth is I’m really not.  It’s proof that we made a big impact in people’s lives.  I hope nobody forgets our work on old school GT and I appreciate those players because without them I’d likely still be sweating over contract work instead of enjoying the freedom I do now.

I do feel bad if someone I replied to on Twitter gets a lot of GT related stuff they didn’t want though (they probably don’t even know what GT is!), and if I reply it makes it worse, so just tweeting at me (or emailing) directly is the best way to go for a GT related question that isn’t answered here.

Will you give me DLs and make me a mod so I can stop casinos?

No I can’t.  I’m not a GT mod or dev anymore, I’m just a player like you.  

By the way, my own kid would DEFINITELY ask developers of Fortnite for vbucks and skins before even saying hello, so I totally understand why younger players ask me that question.  I can’t ever actually say yes though.

Will you make another MMO?

No plans right now.

Can you try to buy back Growtopia?

No, I mean, even if it was for sale and I could afford it, I don’t want to run it again for the reasons mentioned above.  

Maybe in twenty years ask me again though. :)


Will you be in GT during Player Appreciation Week 2020, whenever that is?

Yep, I’ll be there and using /rgo like a madman so feel free to msg me if you see me!

When will <insert thing> be fixed or <insert thing> be added?

Sorry, I don’t know the answers to that kind of thing.

If I see the @Seth account online, is that you?

In most cases yes, but keep in mind this account may be logged on for maintenance reasons by GT staff. (To stop small locks from decaying, for example)

How were the original mods chosen?

Some were carried over because they were trusted mods in my other game Tanked, some were hand picked by me. In later days they were probably mostly suggestions from existing mods or chosen by Solorian. I can’t remember Hamumu ever picking a mod but maybe I’m forgetting.

Why don’t you wear clothes

Bodies need to breathe! Oh you mean in the game? I had the look (the cape, etc) early on and it just sort of stuck.

Oh, you know what, it was originally for a kind of super-hero alter ego I would use called “Heroman” who would run around punishing scammers but maybe I got tired of changing the clothes and just left it on. That’s why the item’s name is “Heroman Cape”.

Technical questions about servers: Were they run in RAID?

Yes, Growtopia’s server used custom picked super fast PCIe SSD raid arrays. Of course I can’t speak to the current technology state or challenges, only what we were using in 2017 and earlier.

How much did you sell GT for?

Er, I avoid answering this because it might fall within an NDA I signed, not sure. However, I’m pretty sure the number is just sitting there in the Ubisoft 2017 public financials so you might be able to figure it out.

Where did you get the music and sound effects?

Both Hamumu and I used a lot of sounds (and music) from our older games as well as new ones. (Some licensed, some created by us) As for music, Cory Mollenhour was commissioned to do the main Growtopia soundtrack. When characters talk it’s usually Hamumu’s voice.

Is there any update you regretted making? Or wish you could change how it worked entirely?

Not really, but I was worried about the vending machine update. I knew it would fundamentally change the game and giant vending machine worlds would replace “trade worlds” where people communicated more. In the end I guess it did, but it wasn’t necessarily a bad thing.

I was against a global market method of buying/selling items for the same reasons. (that’s a feature that never made it in)

What did Hammu do after and do you still talk?

He’s still making great games, go play Robot Wants It All, it’s packed full of Hamumu-humor goodness.

Yes we still talk occasionally. Well, if text counts as talking, but that’s how we always communicated, we never liked meetings or phone calls!

How’s Cosmo doing, does he still play Growtopia?

He doesn’t play these days but there was a time when he and his friends from school were addicted. That’s never happened with any of my other games.

Over the years “Little cute sweet Cosmo” changed into “studies too much math genius teenager Cosmo” who is great in a different way but I’ll never forget this little guy:

Dang, just teared up posting this pic.

Even more stuff to read

If you got this far, why not go all the way and read this stuff too:

Two guys made an MMO – The Growtopia postmortem (2013)

Seth’s old Growtopia question/answer forum thread (2014)

Unity editor script: No BS Cut and Paste

As far as I can tell, Unity doesn’t offer a way to drag an object around the hierarchy in the editor (moving it to a new parent) without also modifying its local transform to keep the rotation/position in the same final worldspace/orientation as before.  (The default copy/paste options have the same issue too)

Sometimes you absolutely don’t want this and I got tired of using work arounds so made this simple script that adds an additional “no frills” cut and paste option to the editor that can be used instead.

To install:  Download NoBSCutAndPaste.cs and put it in any folder named “Editor” that is in your Unity project’s assets folder.  (The “Editor” folder being in a sub-folder is also ok)

Notes:

  • No multi-select, only works on a single object (which can contain sub-objects)
  • Nothing actually happens until you paste a gameobject (it isn’t actually moved until then)
  • If you paste without an object selected, it will be moved to the hierarchy root
  • Undo doesn’t work for this cut and paste
  • (barely) Tested with Unity 2019.3.2f1

The file above is this:

/*
 
 Name:  Cut and paste without changing shit
 Release Date:  2/25/2020
 Version: 1.0
 Credits: Written by Seth A. Robinson except where otherwise noted
 License: No rights reserved, do whatever with it

 Description:

 In the Unity editor, if you drag gameobjects around in the hierarchy, their local position and rotation will be modified so they end up in
 the same final rotation/world position as they had before.  This adds an option so you can do a "pure" cut and paste without that silliness.

 To use:

 Make a folder called "Editor" somewhere in your assets folder (or a subfolder of it) and put this file in it.

 If you right click a gameobject in the editor hierarchy, you should now see two new options "Cut without changing shit" and
 "Paste without changing shit".  Using those you can move an object without Unity modifying its local transform like it normally does.

 Notes:

- No multi-select, only works on a single object (which can contain sub-objects)
- Nothing actually happens until you paste a gameobject (it isn't actually moved until then)
- If you paste without an object selected, it will be moved to the hierarchy root
- Undo doesn't work for this cut and paste
- (barely) Tested with Unity 2019.3.2f1

 www.rtsoft.com
 www.codedojo.com

*/

using UnityEditor;
using UnityEngine;

public class NoBSCutAndPaste
{
    static GameObject _tempObj;

    [MenuItem("GameObject/Cut without changing shit (Shift-Ctrl-X) %#x", false, 0)]
    static void CutWithoutChangingShit()
    {
        var go = Selection.activeTransform;

        if (go == null)
        {
            EditorUtility.DisplayDialog("Woah!", "First click on a gameobject in the hierarchy!", "Ok");
            return;
        }

        var s = EditorWindow.focusedWindow.ToString();

        if (EditorWindow.focusedWindow.ToString() != " (UnityEditor.SceneHierarchyWindow)")
        {
            EditorUtility.DisplayDialog("Woah!", "Don't use the 3D window, click on the gameobject in the hierarchy tree instead before doing cut/paste.", "Ok");
            _tempObj = null;
            return;
        }

        _tempObj = go.gameObject;

        Debug.Log("Cutting" + _tempObj.name+ ", now choose Paste without changing shit");

    }

    //This part by Jlpeebles taken from https://answers.unity.com/questions/656869/foldunfold-gameobject-from-code.html
    public static void SetExpandedRecursive(GameObject go, bool expand)
    {
        var type = typeof(EditorWindow).Assembly.GetType("UnityEditor.SceneHierarchyWindow");
        var methodInfo = type.GetMethod("SetExpandedRecursive");

        var window = EditorWindow.focusedWindow;

        methodInfo.Invoke(window, new object[] { go.GetInstanceID(), expand });
    }

    [MenuItem("GameObject/Paste without changing shit (Shift-Ctrl-V) %#v", false, 0)]
    static void PasteWithoutChangingShit()
    {

        if (_tempObj == null)
        {
            EditorUtility.DisplayDialog("Woah!", "Nothing to paste.  Highlight an object, right click, and choose 'Paste without changing shit' first.", "Ok");
            return;
        }

        if (EditorWindow.focusedWindow.ToString() != " (UnityEditor.SceneHierarchyWindow)")
        {
            EditorUtility.DisplayDialog("Woah!", "Don't use the 3D window, click on objects in the hierarchy tree instead before doing cut/paste.", "Ok");
            _tempObj = null;
            return;
        }

        var go = Selection.activeTransform;
        if (go == null || go.gameObject == null)
        {
            Debug.Log("Pasting " + _tempObj.name + " without changing its local transform stuff.  (Pasted to root as a gameobject wasn't highlighted to parent it to)");

            //Move the object to the root
            _tempObj.transform.SetParent(null, false);
            _tempObj = null;
            return;
        }

        Debug.Log("Pasting " + _tempObj.name + " under "+go.gameObject.name+" without changing its local transform stuff.");
        _tempObj.transform.SetParent(go.transform, false);
        _tempObj = null;

        SetExpandedRecursive(go.gameObject, true);
    }


    /*
     //In theory this would grey out the paste option when it wasn't valid, but due to Unity weirdness it only works in the "GameObject" drop down, not the right
     //click context menu on the hierarchy.  Better to not have it on as it just looks like it doesn't work when using from there.

    // Note that we pass the same path, and also pass "true" to the second argument.
    [MenuItem("GameObject/Paste without changing shit (Shift-Ctrl-V) %#v", true)]
    static bool PasteWithoutChangingShitValidation()
    {
        // This returns true when the selected object is a Texture2D (the menu item will be disabled otherwise).
        return _tempObj != null;
    }

    */

}
 

Magnavox Odyssey 2 Atari Joystick mod

$179 in 1978. That’s $705 in 2019 money.

Meet the Odyssey 2

It’s an old ass game console from 1978 I recently picked up from Ebay. It even has a built in keyboard for some reason.

In the first real “console war” it placed third out of well, three, ending up behind the Atari VCS and Intellivision despite some neat addons like a voice module.

It’s famous for hosting the game K.C Munchkin, a pacman-like game that was forced off the market for being a clone in a giant lawsuit.  However, looking back at it, it played different enough that this really shouldn’t have happened. Sorry K.C, you deserved better.

I buy old systems like this and try to fix them up because it’s a cheap and fun way to learn electronics and gaming history.  Oh, and you get to play your homework!

Care for a game of golf? It’s actually not bad. Your golfer gets mad when he hits trees.

For this one I did a composite video mod as that’s a huge improvement over the original noisy RF that requires a US TV tuned to channel 3 or 4.

The joysticks it came with were completely broken – unfortunately finding replacement controllers for old systems is quite difficult and fixing them to work like new almost always requires new membranes or other parts that are not made anymore.

The solution? Wire it up to accept a standard Atari 2600 controller!  It’s kind of a ubiquitious standard that’s compatible with the Commodore 64 and believe it or not, Genesis/MegaDrive controllers.  I got the idea from ArcadeUSA’s youtube video but I did it a slightly different way so I could also use the original controllers as well.

The electronics for the Atari 2600 controllers are the same but the pinouts are different so you can’t plug them directly in. I used what I had on hand,  a “6inch DB9 Female Port to Dual DB9 RS232 Male Serial Y Splitter Ribbon Flat Cable” for the adapter.  $4 from ebay, I cut off the parts I needed with scissors.

Click for the big version.

I soldered at the pins below the DB9 jacks, easier there.  Above is a labeled picture that might save somebody time.  Notice that the player 2 wiring is SLIGHTLY different from the player 1.

It’s ugly the way they stick out and will surely break soon. But it works great!

Conclusion

It would probably be better to just make DB9 to DB9 adapters to fix the pin differences (could unplug them when not needed), but I didn’t have the right stuff handy.  Someone should make a simple circuit board to do both ports together, something that looks like this useful  joystick port toggle I got for my C128.

Thanks Seth, something I didn’t care about.  When will you actually make a game

Yeah, yeah.  We’ve been working on stuff behind the scenes and will soon be upgrading RTsoft to a new HQ in Kyoto.  The idea is to be a real (well, slightly more legit than now anyway) game studio as well as a sometimes kind of public hackerspace/cafe, more later.

Door game source code release – RTReader

Many moons ago I made a door game scripting engine that read “ref files” (short for reference, I think) to control  what the users see.  Later, this reference file system was used as the base for LORD 2: New World.

Someone asked me today for the source and as I don’t think there are any legal entanglements with releasing this, here ya go.  (About the LORD/TEOS/LORD2 source, I’ve been trying to buy back the rights forever so I can legally release them too)

RTReader V??? source code (probably incomplete and can not actually be compiled, written in Turbo Pascal 6, requires Scott M. Baker’s door driver)

Note:  Does anybody perchance have the RTReader door game .zip file I distributed back in the day?  I think it was a weird kind of “RTsoft company update and virtual interview with Seth” thing, please drop a comment if you know where to find it, thought it might be good for a laugh. My google-fu failed.

UpdateHere is the RTReader thing I was talking about (Thanks Jay).  I’ve packaged it with DosBox, just click play.bat and it will run.

It’s just as hilarious and bad as I expected, complete with tons of misspellings and random things such as a virtual interview and a “game” that ends with a madlibs style story.   It calls itself “Bloody Claw Newsletter Issue 2 from June, 1995”.  So there must be an “issue 1” out there somewhere?  Did an issue 3 ever make it out the door? Hrm.