Here's my attempt at explaining things:
I'm faceing the next problem and may
be its me, but I can't find proper
guides to set up a domain and
nameservers for a dedicated box.
There are three entities in this scenario. The registrar, the DNS host and the hosting server. Whois.com is your registrar. They registered the name for you. Then you need someone to host the DNS records for you. Apparently, Whois.com also does that for you (see under "DNS Management").
I have a domain, at whois.com, and a
Dedicated server at Reliable Hosting
Services, the server has 5 IPs, I know
that I need 2 of them for the
nameservers.
Okay, cool. However, you don't need two of those 5 IPs for nameservers. Whois.com already is your DNS host. So when you look in your whois.com DNS control panel, you then choose to forward your domain to the IP(s) that you want. Technically, you could forward your registered domain to Microsoft.com's servers if you want.
Right now, my domain at whois.com is
using nsX.whois.com nameservers
Excellent. As it should be.
and it has 2 child nameservers:
ns1.mydomain.com & ns2.mydomain.com
pointing to those 2 IPs from my
Server.
Wait a minute, that's not right. As far as I can tell from your scenario, you're not hosting your own DNS servers. Unless ns1.mydomain.com and ns2.mydomain.com are DNS servers that you've set up running BIND or something. What you want is for nsX.whois.com to be your authoritative name servers. From there, you create records like A, CNAME, TXT, whatever you like from your whois.com control panel. For example, an A record that points to the IP of your Reliable Hosting server.
I still cannot set that domain as my
main server domain since it says:
To map an IP to a domain, the domain
must ping to the same IP, otherwise,
the domain will stop working. The
domain you are trying to map this IP
to, doesn't resolve back to the IP,
and so it cannot be set as the default
domain for the IP.
Which system is saying that error? Whois.com's DNS management console or Reliable Hosting? Mapping an IP to a domain is a PTR record. For some reason, that message isn't making sense to me.
Well and I'm stuck on those steps,
whats next to have my nameservers
working and my main domain assigned to
my server?
Regardless of any weirdness that's going on now, you ultimately just want your Whois.com name servers to be authoritative and then create records in Whois.com's DNS management control panel that point whatever A records you have to the IP of your Reliable Hosting server.
All will be well in the land.