No, just knowing the phone number is not enough for a person to track your phone. The phone companies can, and anyone who gains access to cell towers can, but not an average person.
However, smartphones use wifi, and that can be tracked, and in a growing number of cases is tracked. Grocery stores track people in their stores, recognise returning customers, etc. But to track the person, you need to have a "sensor" physically close to the person, so you can't track them out in the world. Russia installed sensors all over cities to attempt to be able to track individuals, but with limited success.
To track someone out in the world without being near them requires access to the cell network, or to install something on the device.