The problem is that tokens like
| aren't processed/expanded by the
ls program. Since the tokens aren't processed,
ls is look for a file or directory literally called
~/org, a file or directory literally called
*.txt, and a file or directory literally called
| head -5. Thus the error message you received about `No such file or directory".
Those tokens are processed/expanded by the shell (like Bourne shell /bin/sh or Bash /bin/bash). Technically, interpretation of the tokens can be shell-specific, but most shell interpret at least some of the same standard tokens the same way, e.g.
| means connecting programs together end-to-end to almost all shells. As a counterexample, Bourne shell (/bin/sh) does not do
~ tilde/home-directory expansion.
If you want to get the expansions, you have to get your calling program to do the expansion itself like a shell would (hard work) or run your
ls command in a shell (much easier):
/bin/bash -c "ls -t ~/org *.txt | head -5"
(call-process "/bin/bash" nil t nil "-c" "ls -t ~/org *.txt | head -5")
Edit: Clarified some issues, like mentioning that
/bin/sh doesn't do