Here's what I got. In 2D, assuming a is the acceleration, d is the direction, p is the inital point and q is the final point:
speed * d_x = (q_x - p_x - 0.5 a_x t\\^2) / t
speed * d_y = (q_y - p_y - 0.5 a_y t\\^2) / t
Solve the y-equation for speed: speed = (q_y - p_y - 0.5 a_y t\\^2) / (t d_y)
Sub into the x-equation, cancel the 't' in the denominator and multiply by d_y: (q_y - p_y - 0.5 a_y t\\^2) * dx = (q_x - p_x - 0.5 a_x t\\^2) * d_y
Solve for t: t = sqrt((q_y - p_y) * d_x - (q_x - p_x) * d_y) / (0.5 * (a_y * d_x - a_x * d_y)))
Then plug that t value back into the speed equation to get the speed.
I note that the expressions like (a_y * d_x - a_x * d_y) look rather like the z-component of a vector cross product, so perhaps there is a way you can generalize this to full 3D easily by using magnitudes of cross products of the vectors involved, rather than having to project / rotate it into 2D.