When counting you ignore case, but when building the result string you don't.
Also, given == "" is not how you compare strings in C.

Hi! While I was looking at my solution, I changed something not important and it said all tests are passed and accepted my solution; however, after I tried the same solution it accepted, it gave error at the 7th test case again as usual.

It might be an issue or I was lucky. Nevertheless, I have no clue what is going on.

Thank you so much for your feedback!!! I've learned a lot!!

In the random tests arr2 is the expected solution, arr is the result of your DuplicateEncoder function.
!strcmp(arr, arr2) is false if the two don't match.

I just submitted my solution successfully, and also didn't see an obvious error in the tests.
If your problem persists, please post your solution here (and make sure to mark it as "having spoiler content").

There are several problems with your approach:

• malloc(strlen(str)) doesn't allocate enough space for the string and the terminating '\0'.
• When comparing a p[i] and p[j] you only convert the first one to lowercase.
  DuplicateEncoder("aA") and DuplicateEncoder("AA") should both return "))".
• If a character appears only once *(res + i) accesses unknown data, possibly even a ')'.

Also, why the complicated memset(res + i, ')', 1); instead of res[i] = ')';?

Did you modify the argument instead of allocating space for the result?

In C:
The expression !strcmp(arr, arr2) is false.

Still, I get all the test cases correctly. Is there something wrong?