You may have commented another version of this solution, but I do not see how primes are recomputed. First, the upper bound is computed, then all primes until the upper bound.

Prime_factors is linear in the values of the elements, times the log(n_elements) insertion time into the hashmap times the number of elements - this risks scaling as n^3 - which means it will be slow, especially on arrays of many large numbers

I was throw off by the set. You create the divisors in order anyways, so I don't see why you want to be returning a BTreeSet over a standard Vec (except perhaps to express that the result is sorted?).

Also, you are recomputing a lot of primes in is_prime, pregenerating all the primes up to the upper_bound in get_all_divisors takes this from worst case O(N^2 sqrt(N) ) to O(N sqrt(N))

Otherwise, I like the functional style, I really need to practice using it myself.