No, you cannot just use the IP address as a nameserver.
To solve the apparent chicken-vs-egg problem, you need glue records inserted into the parent zone. For example.com., the parent zone would be .com. Your registrar must do this..
So, lets say for sake of argument that your VPS's IP address is 10.1.2.3, and you've got a secondary nameserver running on 192.168.168.192..
In your zone file served up by bind, you might have something like this.
example.com. IN A 10.1.2.3
example.com. IN NS ns1.domain.com.
example.com. IN NS ns2.domain.com.
ns1.example.com. IN A 10.1.2.3
ns2.example.com. IN A 192.168.168.192
Then, in addition to providing your registrar with ns1.example.com. and ns2.example.com. as the nameservers, you will need to provide the registrar with IP addresses that correspond to those nameservers. The registrar will then pass that info on to the registry.
Verisign (they operate .com) will then have in their zone file, ie the .com, zone file:
example.com. IN NS ns1.example.com.
example.com. IN NS ns2.example.com.
ns1.example.com. IN A 10.1.2.3
ns2.example.com. IN A 192.168.168.192
with no other entries for example.com. (unless you add more nameservers, or add DNSSEC in which case they would have DS records in the parent zone)
The A records in the parent zone are called "glue records" and they solve the chicken vs egg issue.