Recursion can be fast enough if you cache the results.

The problem may be that when you allocate your char array with c = (char *)malloc(sizeof(char) * (i + 1));, you are not setting the last element c[i] to 0 (or '\0'). Remember that malloc does not initialize any memory, it only allocates it (but calloc will initialize to 0). This can cause a problem with strcmp since it compares the strings up until it reaches 0 on one or both of them.

incredible

lol what

I tried rounding down and still can't figure it out

The tests in Racket are the same as in other languages. Your code is wrong because you think that fractional parts of people give birth which is usually considered as wrong.
Edit: since you "passed" the kata look other solutions that are not wrong.

Your loop has a condition for continuing to repeat.
In order for your loop to stop, something about that condition has to change.
In your loop, you have an if-statement. For one of the branches of that condition, you do not change anything about the condition, instead moving to the next iteration where the same thing happens again.

Suggestion: when things go wrong, use printing to observe what happened.

This is not a well tested kata. Some of the test cases are off by one in Racket. It seems like some expect the population to be greater or equal than the desired one, but some expect it to be greater.

This shouldn't be a solution, although it's very clever how you gamed the system.