God never said "come as you are." When we come to God, we change from what we are. We decide to try to become what He wants us to become. We come to Him because we feel the need for change; to become a person modeled after His son and not "as we are."
However, we should not judge by outward appearances. In 1 Samuel 16:7 we read that man looks on the outward appearance but the LORD looks on the heart. Having said that, a fundamental principle in the Bible is that we always do the very best for God. If our very best is ragged old clothes then so be it. If we have nicer clothing, why do we disrespect God by wearing old worn out clothes? Would such a person wear such clothes to meet an important person or to attend a job interview? No. Why? Because it would show disrespect.
If a person is deliberately wearing old worn out clothes to church as a phony show of humility that is sinful hypocrisy. If they wear such clothing because they are careless that is also sinful. But if they wear such clothing because that's all they have to wear, then God does not care. He cares about the attitude behind the clothing not the clothing itself.