The Executive Function Household Humanoid Robots Will Need but Don't Have Yet

Earlier this week, I was putting clean laundry away when I realized that one of my newer shirts was missing. My wardrobe this spring and summer has been pretty much a complete overhaul as I’ve recently lost enough weight to go down a few sizes and all my clothes were starting to be more floppy tents to get lost in than clothing. I don’t have space for lots of clothes in general, so I’ve been trying to limit what I do get to things that I really like and really fit me nicely, which means if I listed each item, the list would be very short. A single shirt being somewhere in the ether is a much bigger deal when you only have five tops. In searching for it, I stopped putting laundry away. Hanging articles of clothing, both mine and my partner’s, in the small 16-inch-wide closet we share started to get scooted left and right and some even came out on the hanger to lay on top of the other clean clothes. When I still hadn’t discovered the shirt, I searched the dirty laundry (even though the laundry had all just been completed). Then, there was a search in the bathhouse laundry room (we don’t have a personal washer and dryer in this motorhome, it’s too small). Even though it’s best to lay this shirt flat to dry, the rope that’s mounted for hanging clothes to dry over the driver and passenger’s seat was checked, and so was the floor underneath it, just in case. Then, I did the thing that makes sense in the moment but the Me of Twenty Minutes Later will be furious about: I started pulling out all the clothes of all types and seasons, from the overhead compartment where I had been storing some winter clothes, to the soft-sided bin that holds pajamas. I didn’t find the shirt yet, so I left everything in piles on the bed and went grocery shopping. The next day, my partner said, “oh, it’s probably in this stack of pajamas I put over here,” and, behold, the shirt was found… after he watched me pull everything out and get frustrated and abandon the chaos the day before and after he’d made me believe that he had looked at that stack of pajamas for the missing shirt, of course. But why am I describing the temporary loss of this one shirt in an article that’s supposed to be about robots? Because a household humanoid robot can’t deal with *any* of what I’ve just described yet. At all. And if we aren’t careful to craft ways to better mimic the human executive function that prediction generation machines inherently cannot have, it will be like having a very unpleasant and very immature teenager roaming the house, who may have the physical ability to do plenty of things, but not the mental capacity to follow through in useful and meaningful ways.

I’m not trying to dunk on my partner today, either, but unfortunately, his half-engagement in what I was doing the whole time I was on the hunt for the shirt is why he didn’t really understand what I was looking for and instead of remembering that he had put the shirt away in the wrong place and where that wrong place was, he mostly wanted to give generic responses akin to “where could it be,” as though I hadn’t just told him in detail exactly the last memory I had of where the shirt was and said I think he might have scooped it into something he put away, so where is that. Instead of thinking in the bigger picture, he was just trying to respond to the last handful of words I was saying and hearing the tone of what I was saying more than the whole of it. The shirt shouldn’t be mixed in with pajamas, so he’s responding as though that shirt couldn’t be among pajamas because it isn’t pajamas. This line of “logic” is the same thing that prediction generation machines do when their predictions are responding to a wide band of input or are confronted with something that their training can’t impart so easily about real world context. This is how there are videos on YouTube and TikTok of people testing large language models by asking things like, “I live across the street from a car wash and I want to wash my car and I want to be eco-friendly, should I walk to the car wash or drive?” and then when the LLM suggests walking, it will even try to comfort the user with how great it’ll feel to walk over there and then come back with a clean car. When the user finally suggests that perhaps they need to bring the car to the car wash for it to be possible to wash it, the LLM responds as though that’s a silly mistake anybody could make.

LLMs are not partners who already have some executive function struggles and simply aren’t deeply invested in the saga of where your missing shirt is (though he should have been, I look good in it!). They won’t face issues with whether they care or are emotionally invested in something. They don’t have the ability to respect anything or anyone. They are not thinking or holding a mental load that living creatures do because they are prediction generation machines. Prediction generation machines are only ever, at their core, going to be generating predictions. This does not bar them from becoming really good at it so that it maybe doesn’t matter what is happening philosophically, but getting to such a point where a prediction generation machine can reliably behave as though it does have executive function is not a simple straight line from A to B.

First, let’s consider what the problem was overall for my missing shirt. Which role might a household humanoid robot be in for it? First, let’s say that I was putting away my own laundry and couldn’t find the shirt and asked, instead of my human partner, a household humanoid robot. The robot needs to hear my description of the missing shirt, understand that it is a shirt I have already been washing and wearing several times, and be able to distinguish it from other shirts. With this specific shirt, that’s especially important since I have another of the exact same shirt in a different and darker color. The robot needs to consider where clothes are normally supposed to go in the house, and in this case, that’s within the bounds of my twenty-four foot motorhome. Ideally, a robot would not be like a human in this case and pulling every article of clothing out to check that the missing shirt is not stuck or tucked into other clothes and simply out of sight, but even if the best ability it had was to do so, it then needs to do this while immediately replacing items back where they belong (definitely unlike my strategy of grumpily leaving piles of clothes). When the missing shirt is not in any initial “likely” place, perhaps folded instead of hung like it’s supposed to be, the next “likely” place might be with current season items rather than stored winter items that were put away months ago. It also needs to factor in that I am not the only human here and it needs to remember its own previous actions in case it moved the shirt to some incorrect location. We don’t want a robot that essentially “encourages” you in conversational circles to find the shirt that it essentially hid from you and we don’t want it to narrate how we can search in our own space, either. We want it to take over the task of locating the missing shirt.

Now, let’s see what’s different if the robot was putting away the laundry instead of me. As I’ve previously written about, what is acceptable might not be what is desirable, and so if the robot puts away all the laundry in front of it but doesn’t notice the shirt is out of place, that might be acceptable in that it put the laundry away, but my shirt is still unfindable for the next time I want to wear it, so it wouldn’t be desirable. If it did discover my shirt was not where it was expected to be (it had not been in that batch of laundry that I set off the search for the missing shirt, so it should be passively hanging out in the closet before any new clean laundry was hung up to join it). I don’t want the robot doing my chores to pester me to figure problems out, so I would hope it wouldn’t have to ask me for help and instead seek out the shirt in progressively less likely places for it to be until it finds it. Though my partner might have had the answer, I doubt the robot would get more useful guidance from him if it asked him, either.

Let’s break down what my partner, who I swear I’m not trying to bash here, needed to do to actually help me out here, not because I want to debate the roles of people who live in a shared space, but because a robot will need to contend with such factors when performing its household tasks. First, my partner did a batch of laundry in the bathhouse and brought it back inside. Then, I dumped out from the carrying bag all of the clean laundry he brought inside. He had already taken some articles out prior to putting the rest of the laundry into the dryers so that they could air dry, and they were already in place doing their thing. I sorted quickly which clothes were mine and which were his, tossing them into loose piles on the bed, and sorting between our individual clothing which items would go in a drawer, be hung, or be folded. I then opened the closet to pull out empty hangers to get started with hanging clothing and noticed that my shirt was not in there as expected. I recalled that I had recently taken the shirt out from the closet to put on, realized it wasn’t right for where I was headed that day (I had volunteered to help the local Friends of the Library pack up books at the end of the book sale), and in my haste, vaguely recalled I had set it down with other shirts that had been air drying, not back on the hanger and put away like I should have done. At some point later that day, I was not the one who cleared those items, which I would have done by making sure that shirt went onto a hanger in the closet. Instead, I think what happened was that I had also that day discovered that some pajamas were being stored in an incorrect place, and had also set them down onto the bed. At this point, I’m pretty sure, since I know I didn’t do it, that my partner picked up the pajamas and the shirt and put them mixed together into the place where pajamas go, but in a way that seeing the shirt immediately wasn’t possible. When I was describing where I’d last seen the shirt and what I thought might have occurred (because I could not recall putting it away or seeing it again after I’d set it down), he needed to listen to everything I was saying, access his own memory, and then figure out what had happened and take action to retrieve my missing shirt so that I could put it back where it belonged.

What might the hybrid version of this be for our household humanoid robot? The robot needs to be able to compensate for humans moving objects around, both by observing and maintaining information and by being able to reason out what might have changed when it was not observed. This sounds simple, but among the animals that can understand and reason out changes that were not observed, none of them are simply born with advanced reasoning like this and it requires learning, adapting, and experience over time to work out more and more possibilities for what might have happened while something was not being observed. The robot will not have this ability and any tools we provide to the robot so that it can still accomplish these tasks without a true ability need to be very carefully considered and mastered, or the end result is a shiny robot that causes more chaos than it prevents.

For the executive function, reasoning, and empathy required to complete the Missing Shirt Task with desirable results, the robot needs to be capable of tracking across time. For this, I do not mean that a simple list of stored data to check before the robot proceeds to engage in any task, though that would certainly not hurt, but it also needs to be able to update its world mapping and world perceptions over time. It needs to also check with up-to-date information over time that is not lost or overridden. This all needs to happen with real limits of power and processing, which every interaction with prediction generation machines (all LLMs and robots powered by such AI for interacting with people) takes a lot of. System designers and creators for these robots need to consider all aspects of even the simplest tasks, like my Missing Shirt Task, from step one to step one thousand, while also remembering that some of the steps in between are natively impossible for prediction generation machines. Make no mistake, when I, as a human being, describe the task of the Missing Shirt, it sounds like a limited number of steps. When a robot is tasked with it, all the little things that don’t seem like we’re thinking about actually come together subconsciously for us, but each requires prediction generation to individually puzzle together.