Here's a paper on building intrusion detection into OpenWRT. The paper describes the need to limit the signature set due to memory limitations.
This might be worth trying again. Maybe you could get better mileage with something like a WRTSL54GS which has more memory? There's also a lot more features/software around to hook together. Any takers?