Working at a software vendor which provides an SDK for its products, I learned quite a bit about APIs and how to implement them. Despite the global move towards webservice interfaces to connect components of different languages, a clean C interface can still provide a superior alternative.

Some of the reasons why a C interface still is a good alternative are:

• C interfaces are fast.
• C interfaces are cross-language compatible.
• C interfaces can have low footprint.

C APIs can have low footprint

A C API is a simple dll/so file. It's as large or as small as you want it to be. It's as cross-platform as you make it. You can cross-compile it on your dev machine and then deploy it on you tiny MIPS router box with 32MiB of RAM. Try that with a Java based SOAP framework. A C interface needs no sockets, no complicated SSL handshakes, no expensive XML/JSON/whatever parsing. It is just a simple call executed by the CPU.

C APIs are fast

Again, a C interface comes in the form of a dynamic library (dll or a so file). When the application loads a dynamic library, the dynamic linker puts the code of the image into the address space of the calling application. The calling application can then call the API functions via function pointers (either manually or through the GPT). Compared to the marshalling/unmarshalling that is required between webservice caller/callee this is multiple orders of magnitudes faster.

C APIs are cross-language compatible

While it's hard to call from one interpreted language into code from another, you can call from most interpreted languages directly into C.

In Java there is JNI which is a bit complicated but still does the job: The JVM execution thread stops running and jumps into your C code. Type conversion happens in the JNI glue code you have to write in C/C++. The JNI even allows you to call code from the JVM in C/C++ (although I don't consider this a good idea, except for data interchange at the interface).

In C# the compiler/runtime provides PInvoke and the DllImport attribute to do straight forward calls into dll functions.

In Python there are various ways to call C: You can write Python extensions manually, you can generate python extension code with Cython or you can call into given dlls with ctypes.

In Perl there also are various ways to call C. Google for Perl XS or Inline::C. While not having used them myself I guess it won't be too hard.

In Go there is Cgo.

In Ruby you can write Ruby extensions in C.

The list goes on.

And if you're fed up with writing these wrappers manually, you can try SWIG which will generate a binding with your code for various target languages, including the languages mentioned above.

Now to the contrary: If you have code in any of these high level / interpreted languages and try to call it from another of these languages you will go through big trouble as you will have to instantiate the corresponding virtual machine (be it the JVM, the CPython interpreter, the .NET runtime, ...) inside or outside your current process and then figure out a way to pass calls between them. Your best bet may be to create a C interface to your non-native code and then call that from your target language. No fun.

Best Practices For C APIs

Now when designing C APIs there are a few things you can do to keep your interface

• easy to wrap in foreign languages and
• binary stable, so users can replace the dll/so without re-linking (commonly called ABI stability).

Avoid structs

C comes with a few caveats which are non obvious even for experienced programmers. One of those is padding of structs. Say you have a struct

typedef struct color_t {
  char r;
  char g;
  char b;
} color;

From looking at it you could assume that sizeof(color) == 3. But that's not the case. The compiler is free to expand the size of the struct to basically any size it likes. On 32bit platforms it will usually be a multiple of 4 bytes but it depends on the specific compiler options and #pragma pack switches in the header.

Closely related to padding is alignment. To improve the performance of accessing the members of the struct, the compiler can chose to pad the individual data members in a struct so that each of the members is aligned to a specific boundary (again, mostly 4 bytes on 32bit and 8 bytes on 64bit platforms). So between color.r and color.g there may be 3 or more unused dummy bytes. (In reality, there probably won't be padding between char elements, but between elements of different size, there probably will.)

Now when you want to pass structs through a C interface, you have to make sure that both side agree to the same padding and alignment rules. This is a non-trivial problem.

So, my advice is to basically avoid structs in C interfaces. This will save you a lot of trouble. As a replacement, you can work with handles. Everytime you want your user to create a complex struct-like data transfer object, you should provide him with

color_handle make_color(char r, char g, char b);
void remove_color(color_handle);

functions. color_handle would be a typedef void* color_handle;. To access the individual members, you can provide additional char from_color_get_r(color_handle) functions. This will allow you to fully control the implementation details of the color struct and change it at any time as long as you keep the accessor functions stable. Also this nicely maps to an object orient layer that you might want to put above the C level interface in your target language.

Calling Conventions

Another non-obvious pitfall is the calling convention. Both sides of a C function call have to agree how they exchange parameters on the stack or through registers. The standard calling convention should be cdecl but especially on MS compilers there are some other. Your calling convention also depends on the target platform. Wikipedia has a good article on calling conventions. The point here is, you should controll the calling convention on the library implementation and the higher level language wrapper side so that both sides agree.

Memory Management

Heap memory is a delicate thing. As a mental model, one can think of the operating system as only providing the program with address space and mapped pages. Then the program must figure out a way to manage the pages for allocations of smaller objects via malloc/free or new/delete. This task is accomplished by the allocator, which is usually part of the standard C library. But your caller doesn't have to agree. He may want to use a different allocator for performance or reliability reasons. He might be using a completely different C standard library on his side. And at least on windows, every module in a program has it's own logical heap anyway. When a mismatch between malloc/free happens, disaster is bound to happen. There are two possible solutions to this problem:

1. Let the user provide a malloc and free function pointer to the library, or
2. use whatever allocator you choose but make sure that memory from inside the library is never deallocated outside and vice versa.

Solution number one only works with a new project or a very clean codebase where you can change the allocator with minimal amounts of work. I prefer the more practical variant two. In practice you will have to provide your user with make_handle(...) and remove_handle(...) functions for every object you might interchange. As handle type, you can choose void* or even int if you provide an internal handle-to-memory mapping. You might also want to typedef them on different dummy structs so that there won't be any type mismatch on the user side (see the java JNI headers for an excelent example).

typedef struct color_handle_t { int _; } *color_handle;
color_handle make_color(char r, char g, char b);
void remove_handle(color_handle c);

An additional advantage of this strategy is that you also

• can check the validity of any handle before accessing it,
• change the implementation of whatever the handle represents, and
• you can manage handle lifetime and implement opaque reference counting.

You can think of a handle like the this pointer in a C++ class.

Introspection

Traditional C headers often define constants via the C preprocessor #define directive. These preprocessor defines are not accessible by the target language. Thus they have to be duplicated in the target language. This is duplication of knowledge violates the single-point-of-truth programming rule and is a code smell. I prefer to provide

int resolve_define_i(const char *define, int *out_value);
int resolve_define_s(const char *define, char *buffer, size_t *buffer_size);

functions which maps the defines given as C strings to their int/string value. This can be easily implemented in C using the original header defines and removes the burden from the wrapper implementation in the target language.

You might want to provide more introspection functions:

• int get_version(); to query the version of the library (your API will change, prepare for that early on),
• int get_function_signature(const char *function, char *buffer, size_t *buffer_size); to query parsable notations of function signatures.

Consistent Error Handling

This is not specific to C ABI stability but to component design in general: Provide your user with consistent error handling across all your APIs functions. For C interfaces the following rule has proven to be working:

All functions should:

• either return an error code and affect a call-by-reference parameter to provide the output,
• or return the output and provide an error code to a call-by-reference parameter.
• Choose one pattern and apply it consistently throughout your API. No exceptions.

When designing a handle-based API, you can attach additional error messages to the handles when an error occurs.

Provide A Higher Level Wrapper

When you think you are done with your library/API I suggest you go on and try to build a first higher level wrapper. I suggest using Python ctypes as it is a good reference for understanding ABI level C wrappers. First provide a very thin function style target-level to C wrapper, then build an object oriented wrapper on top. You may notice a few things in your API that are not optimal. Go fix them. When you are done writing that wrapper, use it to build something (e.g. a simple command line tool). You might notice more issues with your API. Only when you're finished with this, you should consider your first version of the API stable and go forward to make it public.

There are more best practices for designing good C level APIs. I consider these to be the most essential. If you follow them, you're on a good way and most importantly: You can improve on it by providing new functions for satisfying new requirements.

As some of you (may) have noticed, I have moved this site to a new hosting provider: myself. You know, running your own mailserver and apache gives you that extra bit more of flexibility, confidence and confidentiality.

The DNS entries were transferred yesterday night and some sites still resolve the hostnames to the default IP of the new registrar but that should be all good in a few hours.

If you should notice any hickups of the mail server, please notice me on a different mail account (e.g. cleeus@gmx.de).

Finally, I finished reading Ross Andersons "Security Engineering", from cover to cover. Not getting much time to spend on reading, I consider this an achievement ;) and definitely well invested time. At above link you can get a free PDF of the first edition. Go, read it (if you haven't already) - it's THE BOOK for the ... well ... Security Engineer!

This post became a rather longish braindump of my view on the current situation in the mobile computing devices market. It's based on what I read everyday in the news and some basic economic terms thrown in. Most of the ideas here are not new. You might have read them somewhere else. In fact I have read most of them somewhere in someone else's braindump but can't find it again for linking.

During the students talks at the Information Rules 1 conference, it became clear to me that the mobile devices market with all it's highly integrated smartphones and tablets looks a lot like the mainframe market of the 70's and 80's.

Back then large companies were busy cranking out features on proprietary platforms with proprietary operating systems, locking users deeply in and taking money out of them. This was possible because they controlled the whole platform stack: the hardware, the operating system, the developer tools and a lot of userland software.

This obviously is exactly what Apple is doing (and others are trying to do) today. They tightly control the hardware, the operating system, the developer tools, a lot of userland software and even the distribution channels for 3rd party software. And they do a good job of monetizing that control.

In 1981, IBM did something that was going to change the whole computing industry (and kill their own mainframe business, which, apparently IBM didn't foresee). They released the IBM-PC and opened the spec so that anyone could build clones. Although IBM didn't fully open it (they left out the BIOS code, expecting that none would be able to clone that), many vendors started to build clones. In a short timespan most PCs built were "IBM-PC compatible" .

What followed was a time of incompatibilities, bugs and attempts to restore lock in to specific operating system stacks on top of the x86/IBM-PC platform. But it was also a time of enormous growth. A huge number of hardware vendors popped up and improved the platform, all being held together by the IBM spec and mostly Intel which moved the internal interfaces (PCI, ATA, AGP, PCIe, SATA) forward. It wasn't all beautiful. Microsoft took over a big market share in operating systems. But this can still be considered an improvement as Microsoft didn't control the hardware.

Today only very few companies are still locked in to the old mainframe machines (a few days ago, NASA announced that they finally got rid of their last mainframe, an IBM Z9). Many are free and use software that at least with a bit of work could run on any operating system on any hardware. And even better: We have a whole bunch of free and open source operating systems which run on a multitude of hardware and pretty much can keep up with the commercial competitors.

Now what's happening on the mobile devices market? Apple has taken a good share with its closed iOS platform and makes the mainframe companies of the past look like hippies. Microsoft is just now really entering the game with its Windows Phone platform and strong partnership (if not to call it an acquisition) of Nokia. For me this looks a lot like control over hardware and operating system, too (although Microsoft still licenses Windows Phone to other hardware vendors). Blackberry covers another segment of the market with a closed hardware and operating system stack which they keep for themselves.

This could really have easily become a costly disaster with only one huge monopolist left in the end. But then there also is Google and Android.

With Google developing, pushing and licensing the Android operating system under mostly open source terms (without strong copyleft), diversity is coming back into the game: On the Android platform, the control over the operating system is very much decoupled from the hardware. It's decoupled so much that most of the times you can put a different flavor of Android on your device.

Now Google recently has acquired Motorola Mobility. This looks a lot like the vertical integration strategy of Apple, Microsoft+Nokia and Blackberry. But really I think it's not (and I am not the first to recognize ). Google is an advertisement platform for the open internet. Any company which creates a closed network/platform where Google cannot enter (distribute it's advertisement) is a threat to it. The massive momentum in the closed mobile platforms is such a threat (as facebook is one, too).

With the Android platform, Google provides a free operating system for device vendors (for them it's just free as in beer) which lowers the marginal costs (per-device) for creating mobile devices. This puts price pressure on all closed platform mobile device vendors and creates a huge number of devices which are not so closed and more open for Google's advertisement. Essentially, Google has to prevent any closed platform to get a monopoly or a substantially large share of the mobile device market in order not to loose its advertisement market there. Yes, Google could try to be the monopolist itself. But by giving away Android, Google triggers multiple positive feedback cycles (more devices - more apps - more buyers - more devices - ...). Closing future versions of Android would slow down these feedback loops and thus is not in the interest of Google. Acquiring Motorola has probably more to do with it's patent pool. Microsoft and Apple are known to sue or threaten to sue Android device vendors with their own patent pools and at least Microsoft is known to receive patent licensing fees for Android devices. By acquiring the Motorola patent pool, Google can use these patents to lower or eliminate these license fees and further push the Android platform.

Google's motivation to prevent a monopoly is good for everyone. They make the closed devices a little more open. But this is not yet the freedom we need and mostly have on the PC market. You cannot switch Android for Windows Phone or iOS on any mobile device. At least I have not heard of any device where this is possible.

The reasons is that there was no IBM-PC moment yet for the mobile devices market. You simply cannot swap out the display, the microprocessor, the wireless connectivity components, ... . They all are so tightly integrated, beautifully connected and interwoven with the operating system and userland software that it's hard for an operating system to run on anything other than it was designed for.

Of course this is not only a business problem but mainly an engineering problem. Components are not yet small enough to allow the specification of flexible and capable physical interfaces which could be used to build component based mobile devices as beautiful and usable as the highly integrated devices of today are.

I don't believe there is much more room for innovation on the smartphone and tablet markets in terms of form factors. A smartphone has to fit into a human hand. A tablet has to fit into two human hands. Both are rectangles, because every digital display is a rectangle. And that's about it. Of course there may be other form factors (implants, wearable, ...) but those will be a totally different thing.

Now what needs to happen for an IBM-PC moment on mobile devices are three things:

1. Hardware components have to become a lot smaller.
2. One big player has to make a good spec.
3. A critical mass of standardized devices and components have to be built and sold.

I sincerely hope that this will happen some day.

The good thing is, it's not that unlikely. Once the hardware components will be small enough, device design will not be dominated by engineering problems and solutions anymore. Like a PC case, the phone or tablet could become a case for components. Then everything that is needed will be the spec for developing components and plugging them together. Oh yes, and someone has to start building those machines and components. As with the IBM-PC and Intel, the CPU designer (or a vendor of some other important component which is protected by strong IP rights) could then play the role of the spec designer. When the moment comes, I think we can expect one of the mobile CPU designers (ARM, MIPS, ...) to take that road (because they would get a monopoly, and which company would miss that opportunity).

The bad thing is: It might take some time (or never happen at all). The engineering problems currently seem far away from being easily solvable.

When there is sufficient supply of vertically integrated but differentiated mobile devices (different models with different features), and at the same time the costs of customizing are high then of course market demand for customizable devices will be low. The automobile market works like that. You would never buy some components and plug a car together. Regulatory and technological problems increase customization costs while at the same time the market has differentiated the products enough so that everyone will find a car that fits his needs.

The IBM-PC moment might indeed never repeat, because the IBM-PC was not only an open design, but also a leapfrog at the same time. Unlike most competitors who used 8bit processors, IBM used a 16bit chip in the IBM-PC that was able to deliver considerable more performance. So the advantage of buying "IBM-PC" or "IBM-PC compatible" was not only a minor improvement in a feature here or there, but a big killer feature for the customer.

To repeat that on the mobile market, a similar killer feature may be needed to gain critical mass quickly and get other vendors to adopt. In the prospect of the current engineering problems, sadly, steady evolution towards a automobile-like differentiated but vertically integrated market seems more plausible.

As seen on the PC market, a highly vertically disintegrated market can be very innovative. Open standards are the key as competition no longer has to be about who is setting the standards but about features inside the standard. But finding a player who at the same time is capable of and willing to put a killer feature in use to push an open standard will not be easy. And back then, the IBM-PC moment wasn't really intentional. I believe that had IBM known the consequences, they would not have opened their platform for anyone to clone.

tldr

Mobile devices (smartphones, tablets) are highly integrated products. Vendors control the hardware, the operating system, the developer tools and a lot of userland code. This is similar to the situation on the mainframe market of the 70's and 80's. Back then, the open IBM-PC hardware and software platform broke through the strong lock in effects on those markets. Google's Android is a step into the right direction but not enough. An IBM-PC moment is needed for mobile devices to become free.

disclaimer

It's easy to miss a critical piece when putting these chains of arguments together. If you think I have made a mistake, just drop me a note.

This is a good one on TED:

A.J. Jacobs: How healthy living nearly killed me.