Legal issue surrounding source code.?

I have found myself in a situation in which I am being asked to release source code I had written as a contracted programmer (no actual contract was created, I was contracted in the sense that I was not on taxed payroll, often called independent contracting). I was not paid for much of the time and much of the salary which was owed. A few months after I found a new job, the guy is asking for the source code so that currently working programmers can modify it. There was no mention of back-payment. Can I demand payment for the source code before releasing it to him? Is there a grey area in entitlement and intellectual property in a case like this? Because, in the past, he has payed my minimally, is he entitled to the code then? The monetary amount is roughly 3-4 thousand for something like 200 hours of unpaid development.