Chin is terrified of being held

Hello!
I got a chinchilla, Thor, about two weeks ago. He was very jumpy and terrified when I brought him home. I've spent time just sitting by his cage and what not, and he has gotten fairly comfortable with me. He'll come up to the cage and let me scratch him, climb on my hands, ect.

However, he is terrified of being picked up. He will tremble when you hold him and bite. I've been trying to let him come out of the cage and into my lap once a day so I can pick him up for a few seconds, but he will still fur slip each time. I've been giving him treats and talking to him and he just is terrified still!

I feel horrible that he is so scared. But I feel worse that he cannot come out of his cage in the meantime. The one time I let him out we had the situation where I couldn't handle him to get him back in the cage and it was a disaster. I've been wanting to let him get more familiar before I try it again.

Does anyone have any suggestions on what I should do?
Any help would be appreciated. Thank you