Make hexadecimal and base64 routines behave the same.

The length parameter for the encoding functions is the length of the
binary input, and for the decoding functions it is the maximum size of
the binary output.

The return value is always the length of the resulting output, excluding
the terminating NULL character for the encoding routines.

All functions can encode and decode in-place. The encoding functions
will always write a terminating NULL character, and the decoding
functions will stop at a NULL character.