very clever but if "[(a)]" does not work?

Using while isn't necessary, nor is mutating the input. You can count the number of opening brackets (or closing brackets, if you prefer). If you execute sub that many times then either the string will be empty or there will be an unmatched bracket. You shouldn't use while when the number of iterations is knowable.

I like this! Though I'd suggest not mutating the input.

Chong Voice : "Beautiful, man."

Using gsub! would only apply the subtitution once, so '(({}))' would go to -> '(())'
The while loop continues until gsub returns a falsey value, at which point the string is either empty or the match could not be found.

Ternary at the end isn't needed, just use the empty check.

require 'prime' would be enough