if p points to space, does it mean p is a null pointer?

If p points to anything then it cannot be a null pointer because it pointed to something!

meow meow

thanks for the feedback, there was a function name written wrong. now everything should be fine

@meonu, because x can become very large given large num

In cases ilke this, I mostly put the parentheses in to remind myself about the precedence of the '' (in partucular).

There are several problems with your approach:

• malloc(strlen(str)) doesn't allocate enough space for the string and the terminating '\0'.
• When comparing a p[i] and p[j] you only convert the first one to lowercase.
  DuplicateEncoder("aA") and DuplicateEncoder("AA") should both return "))".
• If a character appears only once *(res + i) accesses unknown data, possibly even a ')'.

Also, why the complicated memset(res + i, ')', 1); instead of res[i] = ')';?

Correct.

val >= 0 is not correct ;-)

And there are other things you need to fix too.

"My code doesn't work" is a question, not an issue ;-)