JavaScript (ES7), 121 117 bytes
x=>(a=b=0,[for(c of x)for(d of'1234')(e=c.charCodeAt()/26|0)==d?a^=1<<d:b^=(a>>d&1)<<d*4+e],f=y=>y&&y%2+f(y>>1))(b)/2
Wow. That was fun. I sketched out an answer idea when this challenge first came out, but it was over 150 bytes long and I didn't want to put in the effort to golf it. I ran across this idea in my notebook yesterday and decided I wouldn't stop thinking about it until I had fully golfed it. I ended up writing out two entirely new algorithms, the first of which ended up several bytes shorter after golfing off around 25 bytes with tons of bit-hacking.
How it works
First we set variables a and b to 0. a is a 4-bit binary array of which bracket pairs we are currently inside, and b is a 16-bit binary array of which bracket pairs are linked together.
Next, we loop through each character c in x, and each char d in '0123'. First we determine what type of bracket c is with e=c.charCodeAt()/26-1|0. The decimal char codes of each bracket type are as follows:
() => 40,41
<> => 60,62
[] => 91,93
{} => 123,125
By dividing by 26, subtracting 1, and flooring, we map these to 0, 1, 2, and 3, respectively.
Next we check if this number is equal to the current value of d. If it is, we are either entering or exiting the dth bracket type, so we flip the dth bit in a with a^=1<<d. If it's not, but we are inside the dth bracket type, we need to flip the eth bit in the dth 4-bit section of b. This is done like so:
b^=(a>>d&1)<<d*4+e
(a>>d&1) Returns the dth bit in a. If we are inside the dth bracket type, this returns 1; otherwise, it returns 0. Next, we shift this left by d*4+e bits, and XOR b by the result. If we are inside the dth bracket type, this XORs the d*4+eth bit of b; otherwise, it does nothing.
At the end of all the looping, b will contain a number of 1-bits equal to twice the desired return value. But we still need to figure out how many bits this is. That's where the sub-function f comes in:
f=y=>y&&y%2+f(y>>1)
If y is 0, this simply returns 0. Otherwise, it takes the last bit of y with y%2, then adds the result of running all but the last bit y through the function again. For example:
f(y) => y && y%2 + f(y>>1)
f(0b1001101) => 1 + f(0b100110) = 4
f(0b100110) => 0 + f(0b10011) = 3
f(0b10011) => 1 + f(0b1001) = 3
f(0b1001) => 1 + f(0b100) = 2
f(0b100) => 0 + f(0b10) = 1
f(0b10) => 0 + f(0b1) = 1
f(0b1) => 1 + f(0b0) = 1
f(0b0) => 0 = 0
We run b through this function and divide the result by 2, and there is our answer.
1Oh, so you're the guy who created CJam?? You owe me for all the answers I lost that got beaten by CJam answers! ;) – kirbyfan64sos – 2015-10-15T22:46:37.003
6@kirbyfan64sos well, you'd better start learning it too if you want to win :) – aditsu quit because SE is EVIL – 2015-10-15T23:26:03.640
9
7~f&? I like this answer already, and I haven't even read the rest of it. – Dennis – 2015-10-16T01:52:10.593