Sometimes netfilter/iptables is just not enough and although there are some extensions and even application layer filters out there, these solutions either don't offer the specific feature you need or have a big overhead.

You could still implement the filter in you application, if you can and that is what you want to do. But, then again, the socket interface hides alot of details from the underlying layers. For example to get access to the securing layer and all fancy stuff inside the IP headers, you need to use RAW sockets or packet sockets. This requires your application (or at least a part of the application) to run as root, which is sometimes also not what you want.

For all these cases, you could go two ways:

a) you could write your own netfilter kernel module

This certainly is the fastest and cleanest way of doing it. But be damned sure not to insert some buggy code in you network stack (on a kernel level)!

b) you could use libipq

This is what I did. Netfilter provides an additional QUEUE target. This target delivers packets into the userspace, where some daemon has the wait for these packets, apply filtering, and then either accept or drop them. Accepted packets then get reinserted into netfilter. This is certainly not the fastest way of doing it, since it requires multiple memcpy's between kernel and userspace, but it hides the complexity and dangers of a kernel module. The mechanism is simple you call ipq_read() which waits for some new packets. If a packet arrives, you get it in a buffer, starting with the first byte of the IP header.

Programming with libipq is very straight forward and a "man libipq" is basically all you need. Add some unix daemon code, some filter code (you can use #include netinet/ip.h to parse ip headers), some commandline parsing and voila - you neat custom packet filter is finished.

Another advantage is, that you don't need to process all arriving packets, but only interesting packets which you can specify in your netfilter rule which uses the QUEUE target. 

Happy packet dropping!