I have been developing Django apps for a lot of time, and almost every project I wrote has some search funcionality. Most time it’s just about searching on a model field but, whenever more than one field or model is involved, I use the reduce approach to search using data in database.

or_queries = [models.Q(**{orm_lookup: bit})
              for orm_lookup in orm_lookups]
qs = qs.filter(reduce(operator.or_, or_queries))

However, when your database is big enough, this approach can lead to long queries. Long queries leads to a poor user experience. What can you do when this happens?. Just use django-haystack and elasticsearch to dramatically improve the user experience when searching.

So let’s begin, first thing is to install some Django packages

Django-haystack

$ pip install django-haystack
$ pip install elasticsearch
$ pip install pyelasticsearch

Add django-haystack to INSTALLED_APPS. Next step would be to create a search_index.py. Let’s say you have a model called Users and you want to search it, your search_indexes.py may looks like this one.

from haystack import indexes
from someapp.models import User

class UserIndex(indexes.SearchIndex, indexes.Indexable):
    text = indexes.CharField(document=True, use_template=True)
    username = indexes.CharField(model_attr='username')

    def get_model(self):
        return User

Don’t forget to include haystack.urls in URLconf. Django-haystak autodiscover will do the magic of finding all indexes. So far we are almost done with django-haystack. You need to configure elasticsearch backend.

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
        'URL': 'http://127.0.0.1:9200/',
        'INDEX_NAME': 'haystack',
    },  }

Let’s provide a convenient template to make the indexing process smooth. You have to include search/indexes/someapp/users_text.txt in your templates loaders. Maybe something like {{ object.username }} will works for you.

Elasticsearch

We need to install elasticsearch in order to use it. This is a super simple task in Ubuntu and Debian. Just pick the deb from download site, and install it using your favorite package manager, locate the config file, and start it (you may want to start it as a service, but that is out of the scope of this post).

$ dpkg -i elasticsearch-1.1.1.deb
$ dpkg -L elasticsearch | grep .yml
<edit config file>
$ elasticsearch -f -D es.config=<path to YAML config>

Now, we must populated the index with some users. We use the rebuild_index management command.

My project doesn’t have lots of transactions to be indexed, so I’m using

HAYSTACK_SIGNAL_PROCESSOR = 'haystack.signals.RealtimeSignalProcessor'

and a weekly full rebuild using the management command through crontab. That may not be suitable to you as real time processor run in-process and your user experience can drop off under load.

Next steps?

Haystack has lots of features I’d like to use, such as autocomplete, queued indexing, “more like this”, and much more. I’d like to benchmark elasticsearch vs solr vs xapian (whoosh is terrible slow for my index size) as a backend too.