should I convert the images into a uniform format or can I just safely
use the same format again?
Quite probably, you will want to crop/resize the image anyway, and possibly add a watermark (which could be visible or not), it is common practice to trace stolen pictures.
So attempting conversion is one way of validating that the uploaded file is a valid image. It goes without saying, but check the extension too. There should be a whitelist of allowed extensions (mainly jpg, jpeg, gif, png). Keep in mind that the file name including extension may be uppercase or even mixed case. Convert the file name to lowercase before extracting the extension.
Do not use the original file name (which could be malicious, contain special characters, or path names like ../../.., or be simply too long for your file system), instead you will probably want to use an ID which could be the ID of the user but it would be better to derive a random string like a UUID, so as not to expose the user ID to the outside world.
Even if a hacker succeeds in uploading a webshell, the directory that stores the images should not allow execution of scripts like php files or whatever. You address this by setting appropriate default file permissions (umask for example) and some additional directives, that will depend on the type of webserver you run.
Obviously hackers will try to bypass the extension check so here is a check list for you. Tampering with the magic number is mentioned, so that alone is not a guarantee that the file will be sound. But the resulting uploaded file may very well be unexploitable anyway.
Is it enough to check the magic number and the file size to ensure an
image is safe to display?
There should be a limit to the file size, that it is up to you to determine. But in itself limiting the file size provides no security. A webshell can be a one-liner.
are there certain formats I should avoid because of possible
vulnerabilites?
The concern is not about the image format, but rather the library you will be using to process that image. For example, the popular ImageMagick package suffered a serious flaw a few years ago. If you're using PHP you could use the builtin functions like imagefromjpeg or the GD library (but it's been a while I haven't touched them so my references may be outdated).
When you've made your choice, do an Internet search for past/present vulnerabilities affecting the particular function/package you intend to use. Even if previous reported flaws have been patched, you still have to make sure that your own version is sufficiently recent (some distros provide old packages, which could still be vulnerable).