If the software is indeed open source,your best bet is to download the source code,audit it and compile it yourself.
Otherwise, there are forensic technics that can assess what the software is doing when running but they are more complex to employ correctly.
In both case,you will need (much) more expertise than can be easily written down in a simple post to reach you goal so if you do not have that expertise, you should consult sooth someone who has.
That said, OSS is usually less prone to that kind of problem since it's easier to expose it by auditing the source code. One way to improve your trust in the software is to download the source code at one moment,wait a couple of month to see if any malicious or problematic section is reported on the projects web page or issue tracker and, if none appears, compile that code and use it.