x86-64 and calling conventions

Well, I have my own compiler, however, I don't as of yet target x86-64.

one of the main reasons:
the calling conventions used in Linux and friends are, scary


Short Description:
arguments are passed in registers, meanwhile, others may be left on the
stack, and there is no particular relation between the registers and the
stack layout.

Beyond Just Putting A Big Damper The Compiler Machinery, It Leaves Many
Ugly Questions:
how does one effectively deal with varargs? (of all things, the convention
still uses regs even for passing varargs).

it seems almost like one will have to dump all of the registers into a kind
of a context, and keep note of particular counts (GPR pos, XMM pos, stack
pos).

how does one do things like hidden argument injection? (useful for many
tasks, such as closures).



, in general, there are all sorts of ugly questions wrt the topic of
actually working with the args list in a first-class manner (beyond just the
task of having to get the compiler to work with all this to begin with).


I am left considering the idea of abandoning this convention for internal
usage, and instead using a different default calling convention. probably,
name mangling would be used to distinguish them, and interface stubs would
be generated to handle external calls.

sadly, this would lead to a big issue, which is that there would be a kind
of a "wall" between the two worlds. firstly, interface stubs would be
needed, and secondly, that function pointers could not be safely passed
across this wall (limiting the usage of callbacks, ).

more-so, these stubs could potentially be limited in terms of what they can
pass along (functions passing raw structs or similar stack-disordering
features may not be wrapable).

so, such a wall could be very problematic (especially in a mixed-compiler
codebase).


now, what kind of convention would I be imagining:
well, basically, it would be a hybrid of the existing conventions, and the
good old 32-bit x86 convention.

in particular, arguments would be passed on the stack (in good old linear
right-to-left ordering).


now, beyond this wall, there is also a possible risk of a performance impact
(both with calls, and in general).

it is also possible I could just "bear with it", and possibly use
stack-flattening/unflattening hacks mostly as a means of dealing with edge
cases.


alternatively, I could use a variant of the MS convention by default (even
on Linux), which is basically about the same (it being fairly simple to dump
arguments to the shadow space, or retrieve them from there).

the task of defeating the "wall" would then be a task for later


any suggestions?
[My suggestion is that if you want people to use the compiler, use the same
calling sequence as everyone else. The code to deal with varargs is ugly,
but I would be surprised if it were very long. -John]

Dave Parker wrote:
(snip)

The calling convention is left-to-right (since I was making my own
calling convention, I figured I might as well fix the old right-to-
left holdover and use the stack the way the stack should be used
left-to-right). Left-to-right means the arguments are evaluated in
the order that users expect them to be. If the function is vararg,
then I push the argument count on the stack last.

That doesn't sound very convenient. I have known for a while now that
ANSI C requires varargs routines to be declared as such so that a
different calling convention could be used.

So far, I haven't known any that actually did that, though.

If you always push the length, one could use it to find the address of
the first argument, and then subsequent arguments.

As far as passing arguments in registers, there is at least one
convention (possibly SPARC, but I am not sure now) where some are
passed in registers, but stack space is still reserved for them. The
called routine can then store them into the stack. As some number of
words are kept in the registers, in some cases a double variable will
be half in a register and half on the stack. Storing to the stack is
a convenient way to get the two back together again.

-- glen

glen herrmannsfeldt wrote:
As far as passing arguments in registers, there is at least one
convention (possibly SPARC, but I am not sure now) where some are
passed in registers, but stack space is still reserved for them. The
called routine can then store them into the stack. As some number of
words are kept in the registers, in some cases a double variable will
be half in a register and half on the stack. Storing to the stack is
a convenient way to get the two back together again.

This is the case for 32-bit PowerPC.

Bolek

Boleslaw Ciesielski wrote:
(I wrote)

>>As far as passing arguments in registers, there is at least one
>>convention (possibly SPARC, but I am not sure now) where some are
>>passed in registers, but stack space is still reserved for them. The
>>called routine can then store them into the stack. As some number of
>>words are kept in the registers, in some cases a double variable will
>>be half in a register and half on the stack. Storing to the stack is
>>a convenient way to get the two back together again.

This is the case for 32-bit PowerPC.

It seems the one I was remembering is SPARC. The LCC book, "A
Retargetable C compiler: Design and Implementation," describes code
generators for MIPS R3000, SPARC, and x86.

For SPARC, the first 24 bytes of arguments go in registers, even if a
double is split in half. Stack space is allocated for them, and also
16 words to store other registers if the register stack overflows.
Floating point values are passed in i registers, so have to be stored
anyway before they can be loaded into floating point registers. It
would have seemed more convenient to pass floating point values in
floating point registers, but that may not be compatible with C,
especially in the case of varargs.

-- glen