An IP address is simply like having your home address on a letter. They can know the area where you live but not more.
A malicious user could try to hack your network but usually, home routers are pretty safe and will prevent any intrusion by default (if you haven’t modified port forwarding and settings like that).
Anyway, random people online are already scanning every IP in the world and already tried to hack in anyone’s home, without success most of the time, and as every website you visite potentially logs your IP already, you shouldn’t be less safe. Pay attention to phishing emails you could receive showing you your IP address as a mean to gain your trust (if the sender tries to impersonate your ISP for instance).
Don’t panic, stay safe.