Web site performance often isn't just about how fast you can serve one page, but also about how many simultaneous users you can handle per server. When you increase the session time, each user stays active in your system for longer, and therefore you can be asking your servers to support more simultaneous sessions. How this impacts your system depends on how you're handling sessions (asp.net has several different session providers you could use).
If you're using database sessions, those tables in your database will now be larger. There will be more disk space and disk I/O on your database, as well as potentially some increased memory pressure. All this could result in session db lookups taking longer.
If you're using in-memory sessions, then of course there is increased memory pressure on your web server, leaving less available for actually serving requests.
The good news is probably none of this matters and you'll be fine. But a more complete answer is that we can't really know from the outside looking in. We don't know what your resources are, or what your load is. As with any change, this stuff can matter depending on where your load is relative to your capacity (and how much information you're keeping in the session) where small increases cross a tipping point that cascade further into the system. This is why it's always important to have good monitoring and alerting tools, and to know your baselines.
