WHY EXECUTABLE FILES ARE OPERATING SYSTEM SPECIFIC.......... ARTICLE 34

ALLOCATION OF THE STACK:

Allocation of the stack requires memory. The memory will be managed by the kernel. So, if you are writing the runtime to allocate the memory, does that runtime code call the kernel functions???? 
YES!!!! It will have to interact with the kernel functions. The kernel will be notified through some interface that the process is requesting the memory for memory for execution. Now think about this... "If you are writing a code that interacts with the Linux Kernel, will this code work on windows???" .

The code which interacts with the Linux kernel's functions are specific to Linux and cannot work on Windows as the functions will be significantly different on both the kernels. The kernel is different for Windows and Linux and hence the runtime code which interacts with the kernel of the operating system should be specific to the OS. This means that runtime code is Operating System specific. Even the linker is OS specific. What ever the linker does, that process is called as the built process. This built process Operating System specific. This is the reason why -exe files which are compiled in the Linux platform does not work on Windows and vice-versa. If we compile and built for one platform, it wont work for the other platform because of the presence of the runtime. 

                                                 Presence of the runtime binds an executable to a specific Operating System. Here if a requirement where we need to compile and built a program which is compatible on Windows and Linux, then we need to build an executable image without the runtime code. Now, the runtime code is necessary for the code to get initialized and without the runtime the code will not run. Now, how do we solve this issue?????

We will remove runtime and make it a software. If we take out the runtime out of the executable and make it a software, then the exe wont contain the runtime. The benefit of this is a single runtime can be shares by the multiple exe's. Virtual Machine of Java is somewhat similar to this concept. All the executables will run directly on that particular virtual machine. With this the executable is independent of the runtime. The benefit with this is we can run the executable in any platform we require which is specific to the OS. Java Virtual Machine is specific to window, JVM is specific to Linux but not the Java code.

In this way we can make the executable independent of the platform by making the runtime code a library. This runtime code will initialize the stack. 

Now what is mean by a library?
How is library going to bind to an executable?
How many types of libraries are there?
We will learn about the libraries in our next article.......................

HOW A PROGRAM EXECUTED WHEN LOADED (PART 2) ..................... ARTICLE 33

In the last article we have seen the exact use of the runtime code. Now, lets implement this runtime code. In this article we will write our own runtime code and append it to the program code. We will tell the linker not to append the default start up files, instead use our own runtime code.

In the below example I am going to write a program with my own runtime code. The program is as follows:-

Here I have written my own runtime code. The actual implementation of the runtime code will be different. What I have done in this program is just calling the functions in the order it happens it default runtime code. The actual implementation of the _fini, _init, and _start will be different. The fini is used form termination of the process and perform a cleanup operation. Now lets execute this program using the Gcc compiler.

Now go through each line of the above image. If you observe carefully, you can see that the compiler throws an error saying that there are multiple definitions of "_start", "_fini".  This error is thrown because the we have already appended the runtime in our functionality and the linker also tries to append the runtime code into our object file which throws an error. Hence, we must tell the linker that it need no to append the startup code or the runtime code. How do we do it????? Lets look into the man pages for help.

In the man pages of the Gcc compiler if you search for the linker options you can see an option called -nostartfiles. Using this option we are telling the linker do not append the runtime code into my object files. Now lets compile the program using this option -nostartfiles.

I have compiled the program with the flag -nostartfiles and successfully executed the program. The output gives you the clarity of how the runtime code is executed. The function calling is very clearly shown in the output.

This is how a program which is loaded into the memory will get executed. The program which we write is inside the main function and the main function calls all the other functions which we write in our .c file. The main function is called by the start function. In my next article we will learn about the stack data structures and libraries.

HOW A PROGRAM EXECUTES WHEN LOADED INTO MEMORY (PART 1).............ARTICLE 32

We have learned about how a program compiles, which tools are internally involved during the process of compilation, what the compiled code comprises of, what is runtime code & what is a PCB..........

Now we will use the understanding of the above concepts and understand how a program will get executed.

A code comprises of 'n' number of functions. Each of these function can have many variable in them. These variables will be allocated addresses when the function is running. Only the global variables are allocated addresses during the compile time. So, the compiler does not know how many variables are present inside the functions. The variables which are inside the function are called the local variables. Before the function starts executing the we need to know how much data the function contains. We need additional memory where we can store the local variables of the function. You cannot determine during the compile time how much memory the local variables require as the local variables are allocated the addresses only during the runtime.

I will give you a better clarity in this concept through an example. Before the example I want to explain about a data-structure called the symbol table.

SYMBOL TABLE:

A symbol table is a data structure used by a language translator such as a compiler or interpreter, where each identifier in a program's source code is associated with information relating to its declaration or appearance in the source, such as its type, scope level and sometimes its location. 

                                       Hence using the symbol table and analyzing the contents of it we can easily check weather the global variable are assigned addresses or not. If the global variables are assigned an address during the compile time, the symbol table will contain the global variable.

In Linux, we have a tool called "nm" which will display the symbol tables in the object file. Lets check the man page of this tool.

The man page clearly says that this command is used to list symbols from object files. I would suggest all the readers to go through the man page in your linux system. Online man pages are also available, just google it.

Now, we will write a small program where we will declare three variables as global variables and three local variables.

In this program we have 8 variables in which four are global variables and four are local variables. In the global variables we have one variable initialized. When I request a symbol table for the executable file of this program we can observe that all the local variables will be assigned an address where as all the local variable will not have any addresses assigned.

This is the symbol table for the above program. You can see three columns in the output. The first column displays the address of the identifier. The second column displays the type of the identifier. Each type of identifier is identified with different letters.

 The third column is a list of the variables in the object files.

 For example: "A" means its the absolute value and will not be changed in further linking. There are many other letters used to identify different types of symbols. Refer the man pages for the further details

In the above symbol table if you carefully observe the variable a, b, c, d are allocated addresses. The variables b, c, and d are not initialized and hence they fall under type "B" which means the symbol is uninitialized and falls in the BSS segment. We have initialized the symbol ' a ' with the value 10 and hence it is of type "D" which means the symbol is in the initialized data section. With this we can clearly figure out that only the global variables are allocated the address during the compilation process.

So, coming back to how an executable file executes, the exe file has only the info about how much global variables we have, and how much code we have. It does not have the information of how many function the code has and each functions has what amount of data until the code starts executing or until the runtime.

Now the runtime code comes into the picture. This is one of the reasons why we require the runtime code. After the code is loaded into the memory and before the program starts executing it will create a block of memory called the stack. Stack is to allocate the data at runtime. Hence the main function of the runtime code or the start up code is to initialize the stack. Executable image only provides the information about what we have declared. Stack is not a part of the executable file.

FUNCTIONS OF THE RUNTIME ROUTINES(_init, _start, _fini):

When an application starts executing it begins with runtime routines. Runtime code is responsible for carrying out following operations.

1) Allocation of the stack segment in the process address space. 

2) Validating and setting up argument stack using command line parameters.

3) Invoking the start function of programming functionality.

When the program starts executing the first function to run is the __init() function. Runtime code means stack. init means initialize.   main() method receives the arguments means, someone must pass these arguments. This runtime code allocates or initializes the stack, receives the command line arguments form the terminal and pass them to the main function. 

                                     The runtime code is implemented to call the main() function. It assumes that the "main" is the first function to be executed. Hence it always start the program with a main. I will try to explain it conceptually first and then I would use an example. The __init() function will call the __start() function. The __start() will call the main() function. When the main() returns the __start() will call the __fini() which would terminate and perform a cleanup operation.

void  __start()

{

    

                                                // in the start rountine

        main();                           // calling the main function.

        __fini();                         // calling the fini function

} 

int main()

{

           printf("main function called by ");

           return 0;

} 

int __fini()

{

     printf("perform the clean up");

     return 0;

}

If you observe the above code, the  function calls are as follows.

__init() ----> __start() ----> main() ---->executes and returns to __start() ----> __fini() ----> executes and returns to __start() ----> __start() returns to its caller __init().

Now you understand why the we use the main function in our programs. Its because, the __start() function is implemented to call the main() function. We can change this general technique in Gcc compiler.

PROCESS CONTROL BLOCK.............ARTICLE 31

WHAT IS PCB(process control block)

We have seen what is meant by process address space. When ever a program is loaded into the memory the memory address space is divided into four regions as we specified in my previous article. This region contains the code of the program the data. Heap is required while execution of the program and the stack is very essential for the execution of the program because the program gets executed inside a stack. The functions we write is executed inside the stack. The data of the functions is executed inside the stack. During the execution of the process if the process requires more memory then heap segment is used to allocate the memory for the process. It allocated additional blocks of memory to the process so that the process executes. Whats important to understand is when the program is loaded into the memory only the data segment and the code segment is allocated. The stack and the heap will be allocated only when the program starts execution.

                                          Now try to understand this concept.. Lets assume the program has been loaded into the memory and now it needs to execute. For execution of the process it requires the CPU time or not??? Yes... without the CPU time the process will not be able to execute. The CPU is a resource and all the resources in our computer system will be handled by the kernel. Hence, we can deduce that for a program to get CPU time it must be registered with the Kernel. Now, when the program which is loaded into the memory it has to  register itself with the kernel to get the CPU time. When the program registers with the kernel, the kernel will create an object specific to this process. This object is what we call as a PCB or the Process Control Block.



PCB is a structure created by the kernel to identify the programs uniquely. This object will contain one variable called ID. Using this variable it will differentiate the processes which are running in the memory. Depending on how many processes are running in the memory, those many PCB's are created in the kernel. CPU will allocate the time to only those processes which are registered with the kernel i.e has a PCB. If there is no Id the kernel could not identify the process and it will not run.

WHAT IS A VIRUS NOW???

If any process somehow manages to run without a PCB then those processes are called as viruses. Usually these viruses will get attached to the existing processes and acquire the CPU time. In this way, the viruses will manage to corrupt many files in the filesystem and make the system slower as it gets attached to multiple processes. Linux is one of the most secure operating systems and this is the reason. In Linux each and every process requires a PCB and viruses cannot get attached to just any  process as the attached program will also require a PCB. In Windows Operating System its not required and hence its easily attracted by viruses. This concept will be more clear when we learn about the threads in Linux. Also keep it in your mind that the above concept is the reason why high definition games are not developed in Linux operating system. We will learn it slowly through my articles.

So, coming back to the PCB it contains a variable called the ID in which it contains a value called the pid. This is the process id of a program. All programs that are getting executed in the memory will be assigned a PCB in which a variable ID is present with a value called pid which is a number with which the process will differentiate every process. The structure of the PCB will look something similar to this.

The image in the left gives a basic idea of the pcb structure diagrammatically. This is just for a basic understanding. The PCB contains all the information about the process. The process number is nothing but the process ID. 

It also contains the current program counter, the number of CPU registers, the memory allocations of a process, the event info, the list of open file descriptors, the program priority information, scheduling policy, resources used by the program... etc.

The kernel maintains the pcb object until the program will terminate. When the program is loaded first an address space is created, and then a pcb object is created for the specific process. Until a pcb is not created its not called a process.

A program becomes runnable only when it becomes a process. When its running on the CPU its called a context. Hence a process could always be not running.

Some might be running and some might be in the wait state. Depending on the CPU availability the process will be in running and wait states.

To access the kernel data structures we have a special filesystem called the proc filesystem. This filesystem will display the kernel datastructures in an ordered format. The below image is how the kernel datastructures are displayed in proc filesystem.

 We will talk about this in detail in our later articles.

KERNEL SPACE AND USER SPACE.............ARTICLE 30

KERNEL SPACE:

The space where the kernel resides is called as the kernel space and the space. It's the area in the memory where the modules and the subsystems of the kernel resides.

USER SPACE:

User space is the area in the memory where the applications resides. Each application has its own memory space i.e its own memory space in the RAM.

PROCESS ADDRESS SPACE:

We call the memory assigned to an address as an address space. There is a term called as the "process address space". Any region assigned to a program is called as the process address region. The particular process address region must contain or comprise of application code.

                                            

                                                 Lets get more specific.... What is there inside an .exe file??? When we say ./a.out we are asking the CPU to load the executable object files image into the memory and execute it. Inside the executable image we have functionality and the runtime code. The functionality is the compiled code which we have written in the .C file and runtime is appended into the object file by the Linker. Now tell me does this source file contains only the instruction??? It also contains the data along with the instructions. Data is nothing but the variable we use in our programs for various purposes. These variables can be inside the function or can be outside the function. The variables which are outside the function are called the Global Variables or we can say the Global data.

                                  So, this global data is to be allocated at the compile time. We have an address for each instruction. When the compiler translates the code into the binary instructions, each instruction it will assign an address. Similar to that for every global variable the compiler will assign an address. So, all global data will have a pre-assigned address offsets. Linker will also relocate them. 

           

                                        Linker decides where the data should be loaded. The Linker also decides where the code should be loaded. Lets think logically for a moment. Inside an executable if there are variable or the data along with the instructions. When the executable is loaded into the memory the data must be loaded into a region which is readable and writable because, there is always a possibility of the variable changing or the data in the variable changing when the program is executing. The code does not change through out the execution. We can summarize it like this........... The data must be loaded into a read/write region and the code must be loaded into a read-only region. The code will only be fetched by the CPU where as the data must be be able to be manipulated during the execution of the program.

                                       Hence for this purpose the memory allocated to the process which is called the process address space is divided into many regions. Each program is given when given an address space for execution of the program in the memory it will be divided into many regions. Each region has its own names. One region is called code and one region is called data. Apart from the above stated address regions there are more two regions. The below image will give you a better idea about the process address space.

In the above image whats referred as the Text segment is called the code region or the code segment and data segment is what we refer as the data region. If you have an idea of the microcontroller programming we will write the data in the data segment and the code in the code segment. The other two regions are called the Stack, Heap and the BSS segment. Each segment or the memory region in the process address space has its own attributes and permissions. 

In the next article we learn about each segments and its purpose.

DEEPER INTO RUNTIME.............ARTICLE 29

We have seen briefly about the Runtime code in my previous article. This article will give you a better understanding about the differences between the relocatable and executable files. Now look at the below screen shots:-

 => This is the relocatable object files main function.




Below is the main function of the executable file.

Now if you clearly observe the above images we can clearly understand that the relocatable object file has only offset addresses.(0: , 1: , 3: ,... ) where as the executable object file has load addresses like 80483e4: , 80483e5: ,........

This is called "Instruction Relocation". The linker decides where to load the above instructions in the executable file. The order in which it has to be loaded into the memory is only decided by the assembler. 

If we observe the above two images there are call instruction in both the object files but, in the relocatable object file its referring to some offset address 38<main+0x38> where as in the executable object file its referring to the load address directly 8048300 <printf@plt>. This functionality of the linker is called function relocation.
Now don't get confused about the tern plt. It's called procedure linkage table. We will talk briefly about it in my later articles.

We have also discussed about what is runtime. It's a compiled set of instruction which is appended by the linker in the executable file. This runtime code is responsible for initialization of the executable file. There are three important initialization functions which Linux provided us.
_init(), _start(), _fini().
The above functions need to be appended to the executable file in order to initialize the object code. Observe the below images. These are the functions which are used to initialize the executable image file.

  Without these functions appended into the executable file we cannot initialize the executable code. These functions are responsible for requesting the operating system for the memory to execute the program, more specifically these are the functions responsible for initializing the function stacks, calling the main function and termination of the program. As we move further we will learn more about these functions. We will write our own _init(), _start(), and _fini() functions and understand how it works.

RUNTIME CODE.............ARTICLE 28

WHAT IS RUNTIME CODE???

We already know the process of  compilation. The GCC compiler will create an object file and at the end the linker will create an executable. I hope by now the process of the compilation is very clear. So, the relocatable  object file and the executable object file contains machine instruction except the below differences.

1: Address

A relocatable object file contains compile time addresses. Whenever the linker creates a relocatable object file for each instruction, it will assign an address. These addresses are called offset addresses. These are not the actual address where the  code will be loaded in the memory. The offset means, its an array depending on how you position your code line by line. Each line will have an offset address. Where it has to get loaded is not yet decided. The sequence is only decided.

The executable object file contains contains the actual addresses. Each instruction is assigned with a load address. This address is machine specific. If my system is 64 bit it will have 64 bit address length. If my machine is 32 bit, it will have a 32 bit address length.

                              This functionality of the linker is called "Instruction Relocation".

2:FUNCTION CALLS

The function calls in the relocatable object file will point to some offset where as the function calls in the executable object file refer to the actual address. It points to the actual load address as assigned by the linker. The call will look something similar to this.

call 80482f0 <printf@plt>------- executable object file

call  11(main + 0x11)----------relocatable object file

                   This functionality of the linker is called "Function relocation".

Executable files contains fully relocated code with runtime. As a programmer whatever we write in a .C file is what we refer as the functionality of the program. Now when we compile the code the assembler creates an relocatable object file. To this file linker will resolve the address and functions which we refer to as instruction relocation and function relocation. Along with this the linker adds a few extra function called the "runtime code".

An executable file is a combination of functionality + runtime. When we say ./a.out we are presenting the executable file to the system to execute it. Now the executable file has to get loaded into the memory. TO load the file in the memory it has to get initialized. TO perform the initialization we require the code called the runtime code.

                               Remember this point always, WHO IS MANAGING THE MEMORY???, the MEMORY MANAGEMENT SUBSYSTEM or we can say the KERNEL. The kernel resides in the Operating system. So OS is allocating the memory for my program to execute when I say ./a.out. So, whenever we need to load a code and request that code to run, that code must have initializations. It must first interact with the O.S. Without initializations the code cannot get loaded nor get executed.

                                       Now we do not know these initialization, the O.S will provide these initialization. It means every time we create an .exe it will include the O.S initializations or application initialization code.

                            In case of Linux three important functions are there

                         1:   _init()

                     2:   _start()

                     3:   _fini() 

These three functions are runtime code provided by the Linux Operating System.

The important concept we need understand is that no matter what language we use the EXE file will have the runtime code. We can summarize it like this

EXE == FUNCTIONALITY(C, C++, JAVA, LISP,... etc)  + RUNTIME.


Note: To get more clarity on this concept please continue reading the future posts