From your description is sounds like you're after setting up RemoteApp rather than the traditional Remote Desktop. So rather than getting a full screen view of the remote server and the apps that are available on it, you want the app itself loaded on your machine so it's visible like any other app on your machine.
Since you already have RD Web Access configured and working you may already have most of the pieces in place, so it's mostly a matter of finding which additional elements haven't already been done.
Most of the guides online refer to 2012 rather than 2012 R2, but the steps are essentially the same between the two versions. I can't find a single guide that I'd really say covers everything, but this post - http://www.riptidehosting.com/blog/how-to-set-up-remoteapp-on-server-2012-r2/ - has links to the best ones I've found plus some others.
Once it's all configured you'll be able to start a published app via the RD Web Access so it opens in its own window (as I say, you may already be at that point), and you can then also optionally connect to your configured collection of apps via "RemoteApp and Desktop Connections" on your client machine to make them easily available locally without the need to use the Web Access site, and once the credentials have been saved locally you can open them in the same way as you would with any other local app.
While you can't pin the remote apps to your task bar like you would a normal app, you can add a new toolbar which links to them (right click the toolbar, go to Toolbars, New Toolbar, then navigate to where the rdp shortcuts are saved and select that folder), and then expand that onto your toolbar to make them easily accessible (they should automatically appear in your start menu).