COPs: Categories-on-Protocols

Abstract

This post describes an interesting extension to the Objective-C runtime.

Project

This was on the Substrate, which was an application server we were developing for Fannie Mae on NextSTEP in 1996-97.

My assignment

I was working on some core parts of the server: the model-driven runtime, configuration manager, request broker etc. The project lead was Jack Greenfield, a brilliant man, who is very much into abstractions, patterns etc. He loves quoting Samuel Hahn's remark Anything you can do, I can do meta. One of specific ideas he brought to my attention was COP or Category-on-Protocol.

I'll explain:

Protocol is the Objective-C word for interface: a pure definition of API. It has neither data nor actual implementations of the API. Classes implement protocols.

Category is a uniquely Objective-C concept. A category is an extension of a class, which cannot add instance variables (fields), but can add methods. Let's say we have a class String. We can add to it a category like this:

@implementation String(Russian)
- (String*) translateToRussian
{
  // Translate this string to Russian and return the result
}
@end

You can send translateToRussian to any String, just as if it were a regular method declared by the class itself.

A Category-on-Protocol is similar to Category, but rather than adding new methods to a class, it associates new method implementations with a protocol. The COP method implementations should be expressed in terms of the methods declared by its protocol. If a class implements this protocol, it automatically acquires all method added to it by COPs. For example, let's say we have a protocol called NaturalLanguageString.

@protocol NaturalLanguageString
- (String*) content;
@end

Now we add an implementation like this:

@cop NaturalLanguageString
- (String*) translateToRussian 
{
  String* string = [self content];
  // Translate this string to Russian
  return string;
}
@end

Once we have such a declaration, all classes implementing the protocol NaturalLanguageString will respond to translateToRussian.

Challenge

COP was a nice idea, but it was not supported by Objective-C. I took it onto myself to implement this concept.

Solution

First of all, I reverse-engineered the Objective-C runtime. Its entrails were undocumented. Turned out that class data structures consisted mostly of dictionaries, where the key was a method selector (const char *) and the value was a pointer to the method implementation. So, to add a method to a class was simply a matter of extending its dictionary with the new (selector, implementation) pair.

In my implementation, a COP declaration was a class (C), which declared that it implemented the protocol in question (P). I wrote a function that would take this class at run-time, figure out what protocol it implemented, find all other classes that implemented this protocol directly or indirectly and add the COP methods to each of them. While doing so, I followed these rules:

• If class X extended, directly or indirectly, class Y, and class Y implemented, directly or indirectly, protocol P, then there was no need to add methods of C as they were already inherited.
• If class X extended protocol P, but had implementations of some of the methods of C, these implementations would not be replaced with the ones from C. This allowed any class to override the implementation of any of the COP methods.

Each COP was registered with the Objective-C runtime to be automatically initialized when the executable started.

I implemented some rather extensive COPs on the Collection protocol: searches, sorting, even execution of queries were automatically available on all collections in the system: arrays, sets, custom collections etc.

COPs proved to be an extremely powerful mechanism and Jack and myself were convinced that the Java programming language would benefit from providing support for COPs. We discussed it with the architects of Java, but they did not want to increase the language complexity at that point. Ironically, they later added to Java support for inner classes, generics etc, which made the language almost as complex as C++. I guess, one cannot escape complexity after all. In light of that idea, perhaps the time for COPs is still to come.

[UPDATE, 2021-01-24: these days "COPs" are ubiquitous. For example, Swift has them as Default Implementations; Java also acquired them as Default methods in JDK 8]