It involves a fair amount of coding. The actual building of an order by query isn't the hard part, it's building the backend params to let you select the "raw" value. If you look at (say) the pre-filter options, where you select a field, for joins we give you two options - Foo, and Foo(raw). The Foo(raw) means you want the "value" rather than the "label". But there is a LOT of coding under the hood to do that.
So we could do that for the Order By, but it's a lot of work. And we have to focus our time on either a) things which the majority of users need / have asked for, or b) features we get paid to code. And this isn't something a lot of people ask for (you are the only one I an remember in the last year or so) ... which kinda leaves option b) ...
-- hugh