Stack is a temporary abstract data type and data structure on the principle of Last In First Out (LIFO). Stack is a container of nodes and has two basic operations PUSH and POP. PUSH adds a node to top of stack and POP removes node from top of the stack.

System Stack

System stack provides a mechanism to allocate memory dynamically for various data associated with a function (or procedure). System stack will store following types of data:

• Local variables

• Return address

• Parameters

• Registers
  

Local variables: All the variables which are created inside a function except static variables are temporary and these will be destroyed after completing the execution of a procedure or after going outside the scope.

Return address: Before CPU transfers execution control to new procedure, it will automatically store the next instruction pointer into a stack.

Parameters: Before calling any procedure, all the parameters either stored in a System stack or it will store in register and this is completely depends on the calling convention of a function and availability of a CPU register.

Registers: Typically, after entering a function, it stores some of CPU registers in system stack and make use of those CPU registers in the current function till it completes the procedure execution.

Win32 calling convention

All arguments are 32 bits when they are passed, return values will be return in EAX and it will be 32 bit. If they are 8-byte structure, then it is returned in EDX: EAX, Larger structure is returned as a pointer in EAX register for hidden structure.

All parameters are pushed to function and stack clean up happens based on the calling convention. Compiler will generate prolog and epilog code to save and restore ESI, EDI, EBX, EBP registers if they are used inside a function. Compiler can create new calling conventions based on the need.

There are different types of calling convention available in Visual C/C++ compiler:

• cdecl: Parameter push happens from right to left and caller will cleanup parameters from stack. This is mainly usefully if a function has variable arguments, in this case only caller will know to cleanup the stack. Although most C function uses this convention. This option is set by default in Visual C/C++ compiler.
  

• stdcall: Parameter push happens from right to left and callee will cleanup parameters from stack. Win32 API uses primarily this convention.
  

• fastcall: First 2 DWORD parameters will be stored in registers (ECX & EDX) and rest will be pushed to stack from right to left. If there is any parameters pushed into stack, then the callee will cleanup the stack.
  

• thiscall: This convention is mainly used in C++ member functions and parameters are pushed from right to left. this pointer is stored in ECX register instead of pushing into a stack. Callee will cleanup the stack.

Code generation for cdecl:

In this example parameters are pushed from right to left and after completing cdeclFunc() caller is cleaning up the stack. Marked with Square Box shows cleanup code. ADD ESP, 10h means 16 bytes cleanup from stack.

Code generation for stdcall:

Caller is not cleaning up the stack in above code.

Marked with square box shows cleanup code. RET 10h means 16 bytes cleaning up from stack.

Code generation for fastcall:

First 2 DWORD parameters compiler is using ECX and EDX registers. Rest of the parameters is pushed from right to left. Caller is not cleaning up stack in above code.

ECX and EDX registers will have 2 parameters data and rest will be popped from stack where ever it requires. In our example we pushed 2 DWORD parameters in stack. RET 8 means it is removing 8 bytes data from stack.

Code generation for thiscall:

this pointer is passed in ECX register.

Callee is cleaning up the stack. RET 10h means 16 bytes cleanup from stack.

Executing a function in High level language will result in following operations internally:

• Push parameters into the stack

• Call the function. This will result in storing return address in stack

• Save and update EBP register

• Allocate local variables in stack

• Save CPU registers what ever used inside a function (PUSH registers)

• Execute code of a function which has some purpose

• Release local variable storage from stack

• Restore saved registers (POP all pushed registers)

• Restore the old base pointer (EBP)

• Pop the return address and transfer control back to caller

• Cleanup the pushed parameters (Based on calling convention)

For example: Let’s say you call a function which is of type cdecl TestFunc (10, 20, 30, 40);

Code generation for calling a function will look like:

Parameters are pushed from right to left: 40(28h), 30(1Eh), 20(14H), 10(0Ah). After executing instruction CALL TestFunc stack will look like following:

In memory window shows the stack dump. Current stack pointer after entering function TestFunc() is 0x0012F21C (Check ESP register in Register window).

In a stack dump you will find following data:

• 0x0012F21C have value of 0x004151d0. This is return address where function should go back after executing function. This address is visible in previous code during calling.
  

• 0x0012F21C+4 have value of 0x0000000a(10). This is first variable which is passing to TestFunc().
  

• 0x0012F21C+8 have value of 0x000000014(20). This is second variable which is passing to TestFunc().
  

• 0x0012F21C+12 have value of 0x00000001e(30). This is third variable which is passing to TestFunc().
  

• 0x0012F21C+16 have value of 0x000000028(40). This is fourth variable which is passing to TestFunc().

After entering a function, it stores EBP register in stack for use inside this function. This register is used mainly to access the stack and its parameters. If it’s using any registers during the execution of this function it will PUSH all those registers. In our example, it is pushing register EBX, ESI, EDI.

Once it retrieves parameters it will execute the function to serve it purpose, releases all the local variables memory from stack if there is any, restores the registers which was PUSH’d earlier like EDI, ESI, and EBX, restore the EBP register which was PUSH’d earlier and return from function. RET instruction will POP the return address 0x004151d0 and transfers control to the caller.