Right Tool for the Job
In one of the projects, our client wanted an auto-complete function in one of the search forms. At first we implemented it with SQL queries because it was the simplest and logical solution. It worked well with small data set containing about a hundred records. The response was fairly snappy - it took less than a second to fetch data and display the results. However, this solution did not scale well with large data set containing several hundred thousands of records. The response time was about half a minute - clearly not acceptable for an auto-complete widget.
The problem had to be solved from a different angle. Rather querying the database with select statements, we used a free text search engine called Lucene via the sfLucenePlugin. Execution time was reduced substantially but it was still too slow. Since Lucene was originally written in Java, we did a quick test to see how well the Java implementation performed. To our delight, it performed very well - results were returned in less than a second.
The challenge was how to make Java and PHP talk to each other. We experimented with a number of alternatives. In the end we chose XMLRPC over HTTP to bridge the language gap. The protocol is well supported by both languages. It only takes a few lines to create an XMLRPC request object, send it to the XMLRPC Java server and get back an array of PHP objects. No home-brewed, string delimited request/response protocol! The architecture also allows the two processes to run on separate servers or have one Lucene server service multiple PHP servers for better scalaliblity and hardware utilization.
What was the lesson we learned? Use the right tool for the job: if a library suck in one language, try libraries written in other languages (or write one yourself in another language).
Tags: integration, java and php, lucene, xmlrpc
