nx1.info | Docker Bullshit

aka Why the fuck is this container still here?

Docker Compose is supposed to make microservices "deterministic" and "easy." That’s a goddamn lie. Especially on edge AI gear, Docker occasionally decides to lose its mind and birth "phantom processes" - zombie containers that refuse to die, survive reboots, and generally exist just to spite you. This is a post-mortem of a total state-management meltdown on an NVIDIA Jetson Orin NX. I’m going to walk through how the Docker daemon decided to gaslight me into thinking a container didn't exist while simultaneously blocking me from doing anything because it *did* exist. The "fix" is basically manual brain surgery on the filesystem because Docker's own tools are useless when it gets into this state.

I. Docker is a Pathological Liar

I was running a watchdog script (run_smartbench.sh) to manage a docker compose up sequence. The idea was simple: if something breaks, kill it all and start over. But Docker had other plans. Every time I tried to start the stack, it would immediately shit the bed and cough up this nonsense:

Error response from daemon: No such container: 042fd7bc7537...

Docker Compose thought the container was there. The Docker Engine thought it wasn't. They weren't on speaking terms, and I was stuck in the middle of their shitty divorce.

II. Chasing My Own Tail (The Logger.py Incident)

At first, I thought I was the idiot. I tried to isolate the services and run them manually with docker run. Turns out, I *was* an idiot—the GUI and Data Manager were crashing because I forgot a logger.py script. I fixed the volume mount (-v /path/to/logger.py:/app/logger.py), thought "hell yeah, we're back in business," and ran docker compose up again. Nope. The original "No such container" error stayed right there, mocking me. This wasn't an application bug; the Docker daemon's internal state was just fundamentally fucked.

III. The Prune-and-Pray Methodology

I tried the "standard" fixes. You know the drill—prune everything and hope the daemon remembers how to do its job.

docker compose down --remove-orphans
docker network prune -f
docker compose up -d --force-recreate

Zero effect. The daemon kept insisting 042fd7bc7537 didn't exist, while also refusing to let go of whatever ghost it was haunted by.

IV. The Paradox of the "Ghost" Container

I ran docker ps -a and, surprise! there it was:

Exited (143) 6 weeks ago

Just sitting there. Mocking me. So I tried to kill it with fire:

matoha@orin-nx-1:~$ docker rm -f 042fd7bc7537
Error response from daemon: No such container: 042fd7bc7537

Look at that shit. "No such container" while it's literally listed in the output of the previous command. I restarted docker, restarted containerd, killed every containerd-shim, I restarted the whole computer and every process I could find. Nothing worked. The ghost container was immortal.

V. Finding Where Docker Hid the Bodies

When the CLI is lying to you and the daemon is broken, you have to go into the basement with a flashlight. Usually, Docker keeps its metadata in /var/lib/docker/containers. I went there and... it was empty.

root@orin-nx-1:/var/lib/docker/containers# ls -l
total 0

For a second, I thought I was hallucinating. Then I remembered this is a Jetson Orin NX. We don't use the shitty internal eMMC; we use an NVMe SSD because we actually want things to work. I checked the config:

root@orin-nx-1:/var/lib/docker/containers# docker info | grep "Docker Root Dir"
 Docker Root Dir: /ssd/docker

Bingo. Docker was keeping its shitty, corrupted secrets on the SSD.

VI. The Final Kill Command

Once I found the actual root directory, the fix was simple: stop the service and manually delete the folder that Docker was too stupid to handle. 1. Stop the Docker service before it realizes what's happening. 2. Nuke the corrupted metadata folder on the SSD:

rm -rf /ssd/docker/containers/042fd7bc753741e23065557633643464bb51c3ebf78baa03b9a8f17ed568ea3a/

3. Turn Docker back on and tell it to behave. After I manually deleted that folder, docker ps -a was finally clean. No more ghosts. The run_smartbench.sh script fired up all 13 containers without a single complaint. The moral of the story? Don't trust the Docker CLI when it says a container doesn't exist. If you're on a non-standard platform like a Jetson, check where your root dir actually is and be prepared to manually delete the garbage that Docker leaves behind when its metadata database shits itself.