Nicely down with the .appendReplacement!

One note: the English alphabet has 26 letters [a-zA-Z] but your pattern uses \w, which has extra chars: [a-zA-Z_0-9].

It doesn't work correctly if you have non A-Z in front of the text.
For "1a2" it returns "2a".

Solution doesn't work if the string contains something other than {}()[]. The current description says that a string may contain these characters, not that it contains only these characters.

s.toCharArray() creates a copy of the original string, so the performance suffers if the function is called often, with big strings.

I would use
return closeStack.empty();
instead of
return (closeStack.empty()) ? true : false;

Please add a test for 9223372034707292160L, that is (4294967296L * 4294967295L) / 2.

Doesn't work for 9223372034707292160L.

mwrum, your solution is not correct either.
The expression (sum + i > Integer.MAX_VALUE) is always false if sum and i are both integers (at least one should be long), because the sum will overflow and become negative.

tri will be -2305843008139952128 for input n = Integer.MAX_VALUE, so it won't return 0.
long tri = n * ((long)n + 1) / 2; would fix the problem but it's error prone. I would just save n into a long first.
Also in the if() you don't need an explicit cast to long.