Chinchilla has no desire to leave cage?

So, I've posted before about my nervous boy, Nimbus, and things have gotten better, he's warmed up to me a little more, but he has no desire whatsoever to explore outside his cage. Free-roaming time terrifies him and he just hides under the nearest thing he can get under. I bought him a harness to get him out of the cage more, walk him in other places around the house that he shouldn't necessarily free roam in... he barked the whole time I tried to put it on him, and once in it, he ran a little, but then just sat stationary looking zoned out and tried very hard to get it off him. I've done this twice now, and he's just a very unhappy boy and I feel like the worst chin mom ever. He just runs away from me now when I open the cage and is very skittish about pets cause he thinks I'm gonna pick him up.

Is it fine that he doesn't want to leave the cage? He lives in a ferret/critter nation 142 (the old, clanky version). It's double one so he has two floors. I provide new chews and things for him when I have the money, and always throw him a toilet paper tube to chew on. It's probably his favourite.

What can I do to help regain trust and make him comfortable?

I'm definitely feeling frustrated and frazzled. :/