NEW NAME. SAME EXPERTISE.
We changed our name! After 14 years of creating award-winning digital products & services, it’s time for a new identity that better reflects the human insights-driven, digital customer experiences we create.
VISIT OUR NEW SITE
NEW NAME. SAME EXPERTISE.
We changed our name! After 14 years of creating award-winning digital products & services, it’s time for a new identity that better reflects the human insights-driven, digital customer experiences we create.
VISIT OUR NEW SITE

AIAIO: Our Blog

AIAIO: Our Blog

The pulse and reviews of Alexander Interactive

Getting Started with Solr and Django

Solr is a very powerful search tool and it is pretty easy to get the basics, such as full text search, facets, and related assets up and running pretty quickly. We will be using haystack to do the communication between Django and Solr. All code for this can be viewed on github.

Install

Assuming you already have Django up and running, the first thing we need to do is install Solr.

curl -O http://mirrors.gigenet.com/apache/lucene/solr/4.0.0-BETA/apache-solr-4.0.0-BETA.zip
unzip apache-solr-4.0.0-BETA.zip
cd apache-solr-4.0.0-BETA
cd example
java -jar start.jar

Next install pysolr and haystack. (At the time of this writing the git checkout of haystack works better with the Solr 4.0 beta then the 1.2.7 that’s in pip.)

pip install pysolr
pip install -e https://github.com/toastdriven/django-haystack.git

Add ‘haystack’ to INSTALLED_APPS in settings.py and add the following haystack connection:

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.solr_backend.SolrEngine',
        'URL': 'http://127.0.0.1:8983/solr'
    },
}

Full Text Search

For the example, we’re going to create a simple job database that a recruiter might use. Here is the model:

from django.db import models
from django.contrib.localflavor.us import models as us_models

JOB_TYPES = (
    ('pt', 'Part Time'),
    ('ft', 'Full Time'),
    ('ct', 'Contract')
)

class Company(models.Model):
    name = models.CharField(max_length=64)
    address = models.TextField(blank=True, null=True)
    contact_email = models.EmailField()

    def __unicode__(self):
        return self.name

class Location(models.Model):
    city = models.CharField(max_length=64)
    state = us_models.USStateField()

    def __unicode__(self):
        return "%s, %s" % (self.city, self.state)

class Job(models.Model):
    name = models.CharField(max_length=64)
    description = models.TextField()
    salary = models.CharField(max_length=64, blank=True, null=True)
    type = models.CharField(max_length=2, choices=JOB_TYPES)
    company = models.ForeignKey(Company, related_name='jobs')
    location = models.ForeignKey(Location, related_name='location_jobs')
    contact_email = models.EmailField(blank=True, null=True)
    added_at = models.DateTimeField(auto_now=True)

    def __unicode__(self):
        return self.name

    def get_contact_email(self):
        if self.contact_email:
            return self.contact_email
        return self.company.contact_email

The next step is to create the SearchIndex object that will be used to transpose to data to Solr. save this as search_indexes.py in the same folder as your models.py. The text field with its template will be used for full text search on Solr. The other two fields will be used to faceted (drill down) navigation. For more details on this file, check out the haystack tutorial.

class JobIndex(indexes.SearchIndex, indexes.Indexable):
    text = indexes.CharField(document=True, use_template=True)
    type = indexes.CharField(model_attr='type', faceted=True)
    location = indexes.CharField(model_attr='location', faceted=True)

    def get_model(self):
        return Job

    def index_queryset(self):
        return self.get_model().objects.all()

Create the search index template in your template folder with the following naming convention: search/indexes/[app]/[model]_text.txt
For us, this is templates/search/indexes/jobs/job_text.txt

{{ object.name }}
{{ object.description }}
{{ object.salary }}
{{ object.type }}
{{ object.added_at }}

Now, lets get our data into Solr. Run ./manage.py build_solr_schema to generate a schema.xml file. Move this into example\solr\conf in your Solr install. Note: if using Solr 4, edit this file and replace stopwords_en.txt with lang/stopwords_en.txt in all locations. To test everything and load your data, run: manage.py rebuild_index Subsequent updates can be made with: manage.py update_index.

If that all worked we can start working on the front-end to see the data in Django. Add this to your urls.py

(r'^$', include('haystack.urls')),

At this point there are at least two templates we’ll need. One for the search results page, and a sub-template to represent each item we are pulling back. My example uses twitter bootstrap for some layout help and styling, see my base.html here if interested.

Create templates/search/search.html
This gives you a basic search form, the results, and pagination for a number of results

{% extends 'base.html' %}

{% block hero_text %}Search{% endblock %}
{% block header %}
Click around!

{% endblock %}

{% block content %}</pre>
<div class="span12">
<h1>Search</h1>
<form class=".form-search" action="." method="get">{{ form.as_table }}
 <input type="submit" value="Search" /></form></div>
<pre>
{% if query %}</pre>
<div class="span8">
<h3>Results</h3>
<div id="accordion2" class="accordion">{% for result in page.object_list %}
 {% include 'search/_result_object.html' %}
 {% empty %}

No results found.

 {% endfor %}</div>
 {% if page.has_previous or page.has_next %}
<div>{% if page.has_previous %}<a href="?q={{ query }}&page={{ page.previous_page_number }}">{% endif %}« Previous{% if page.has_previous %}</a>{% endif %}
 |
 {% if page.has_next %}<a href="?q={{ query }}&page={{ page.next_page_number }}">{% endif %}Next »{% if page.has_next %}</a>{% endif %}</div>
 {% endif %}</div>
<pre>
{% else %}

{% endif %}
{% endblock %}

And the templates/search/_result_object.html

{% with obj=result.object %}</pre>
<div class="accordion-group">
<div class="accordion-heading"><a class="accordion-toggle" href="#collapse_{{ obj.id }}" data-toggle="collapse" data-parent="#accordion2">
 {{ obj.name }}
 </a>
<div style="padding: 8px 15px;">
Company: {{ obj.company }}

Type: {{ obj.type }}

 {% if obj.salary %}
Salary: {{ obj.salary }}

{% endif %}

Location: {{ obj.location }}</div>
</div>
<div id="collapse_{{ obj.id }}" class="accordion-body collapse in">
<div class="accordion-inner">
Contact: <a href="mailto:{{ obj.get_contact_email }}">{{ obj.get_contact_email }}</a>

 {{ obj.description }}</div>
</div>
</div>
<pre>
{% endwith %}

Start up your dev server for search!

Related Items

Adding Related Items is as simple as using the related_content tag in the haystack more_like_this tag library and tweaking out Solr config. Open up solrconfig.xml and add a MoreLikeThisHandler within the tag:

<requestHandler name="/mlt" class="solr.MoreLikeThisHandler" />

Our full _result_object.html now looks like this:

{% load more_like_this %}

{% with obj=result.object %}
<div class="accordion-group">
    <div class="accordion-heading">
        <a class="accordion-toggle" data-toggle="collapse" data-parent="#accordion2" href="#collapse_{{ obj.id }}">
            {{ obj.name }}
        </a>
        <div style="padding: 8px 15px;">
            <p>Company: {{ obj.company }}</p>
            <p>Type: {{ obj.type }}</p>
            {% if obj.salary %}<p>Salary: {{ obj.salary }}</p>{% endif %}
            <p>Location: {{ obj.location }}</p>
        </div>
    </div>
    <div id="collapse_{{ obj.id }}" class="accordion-body collapse in">
        <div class="accordion-inner">
            <p>Contact: <a href="mailto:{{ obj.get_contact_email }}">{{ obj.get_contact_email }}</a></p>
            {{ obj.description }}
            {% more_like_this obj as related_content limit 5  %}
            {% if related_content %}
                <div>
                    <br>
                    <p><strong>Related:</strong></p>
                    <ul>
                        {% for related in related_content %}
                            <li><a>{{ related.object.name }}</a></li>
                        {% endfor %}
                    </ul>
                </div>
            {% endif %}
        </div>
    </div>
</div>
{% endwith %}

Facets

To get our type and location facets, we’ll have to add them to a queryset and pass this to a FacetedSearchView instead of the default one. urls.py now looks like this:

from django.conf.urls import patterns, include, url
from haystack.forms import FacetedSearchForm
from haystack.query import SearchQuerySet
from haystack.views import FacetedSearchView

sqs = SearchQuerySet().facet('type').facet('location')

urlpatterns = patterns('haystack.views',
    url(r'^$', FacetedSearchView(form_class=FacetedSearchForm, searchqueryset=sqs), name='haystack_search'),
)

Then, we can use the generated facets in the search template in the facets variable

{% extends 'base.html' %}

{% block hero_text %}Search{% endblock %}
{% block header %}<p>Click around!</p>{% endblock %}

{% block content %}
<div class="span12">
    <h1>Search</h1>
    <form method="get" action="." class=".form-search">
        <table>
            {{ form.as_table }}
        </table>
        <input type="submit" value="Search">
    </form>
</div>
        {% if query %}
            <div class="span2">
                <h3>Filter</h3>
                {% if facets.fields.type %}
                    <div>
                        <h4>Type</h4>
                        <ul>
                        {% for type in facets.fields.type %}
                            <li><a href="{{ request.get_full_path }}&amp;selected_facets=type_exact:{{ type.0|urlencode }}">{{ type.0 }}</a> ({{ type.1 }})</li>
                        {% endfor %}
                        </ul>
                    </div>
                {% endif %}
                {% if facets.fields.location %}
                    <div>
                        <h4>Location</h4>
                        <ul>
                        {% for location in facets.fields.location %}
                            <li><a href="{{ request.get_full_path }}&amp;selected_facets=location_exact:{{ location.0|urlencode }}">{{ location.0 }}</a> ({{ location.1 }})</li>
                        {% endfor %}
                        </ul>
                    </div>
                {% endif %}
            </div>
            <div class="span6">
                <h3>Results</h3>
                <div class="accordion" id="accordion2">
                    {% for result in page.object_list %}
                        {% include 'search/_result_object.html' %}
                    {% empty %}
                        <p>No results found.</p>
                    {% endfor %}
                </div>

                {% if page.has_previous or page.has_next %}
                    <div>
                        {% if page.has_previous %}<a href="?q={{ query }}&amp;page={{ page.previous_page_number }}">{% endif %}&laquo; Previous{% if page.has_previous %}</a>{% endif %}
                        |
                        {% if page.has_next %}<a href="?q={{ query }}&amp;page={{ page.next_page_number }}">{% endif %}Next &raquo;{% if page.has_next %}</a>{% endif %}
                    </div>
                {% endif %}
            </div>
        {% else %}
            <div class="span6">
                {# Show some example queries to run, maybe query syntax, something else? #}
            </div>
        {% endif %}
{% endblock %}

And we’re done! As I said, check out the haystack documentation for more information. Leave any questions in the comments and I’ll be sure to answer them. Spelling suggestions to come in the next post.

Technology
  • Rishabh Yadav

    I followed the same steps as mentioned above but my query returns no search results.I am using Sqlite3 in my Django Project.

  • Do you see results in the Solr admin console?

  • geoff washam

    Using solr 4.3.0 and haystack 1.2.7 and django 1.5.1

    Helpful tutorial, no real problems running it using sqlite3. I also had to add this line to the generated schema.xml to get solr to work properly:

    thanks

  • kenneth

    Thank you for the tutorial.it helped me a lot,however I’m stuck on one point.

    In trying to index my data on solrn it gives me this error : http 404 error, missing required field: Id.

    How can that be resolved?

  • 伍正飞

    the demo above has a bug:

    def index_queryset(self):
    return self.get_model().objects.all()

    should be

    def index_queryset(self, using=None):
    return self.get_model().objects.all()

    if you query nothing, try modify like this, and run python ./manage.py rebuild index.

  • O

    Nice tutorial.
    After edited schema.xml, restart your Solr server.

  • Andy

    Thanks for the detailed tutorial.

    For at least Haystack v. 2.1.0, search_indexes.py should be updated to have the following prototype:

    def index_queryset(self, using = None):

  • Shireesha

    May you please tell what is the need of

    from local_settings import *

    in settings.py file

    when I run
    python manage.py runserver I am getting error for this import

  • Tim

    @Shireesha

    We use that pattern to hold environment specific settings.

    settings.py has all the settings that don’t change, local_settings.py has things such ad db credentials

  • If you receive this error
    unknown field ‘django_ct’
    unknown field ‘django_id’
    please see this discussion
    https://github.com/toastdriven/django-haystack/issues/878

  • SalahAdDinYusuf

    Hi bro, i’m using haystack with elastich search, and i build the indexes, and y test with django shell, but, when i try view the results in my template, i can’t see anything, :'(
    And, i have other question, i have a index template and a field for search, how can i write in this field and redirect to search template with the results for this searc? as web pages, you know :D

  • larry peng

    Hi,When I using haystack with solr search, I have a question:

    when I have five databases in my project how to search all databases when queries.

    my database config in my django :

    DATABASES = {
    ‘default’: {
    ‘ENGINE’: ‘django.db.backends.mysql’,
    ‘NAME’: ‘{}_default’.format(PROJECT_NAME),
    ‘USER’: ‘root’
    }
    }

    DB_MOD = 4
    for i in range(DB_MOD):
    db_dict = DATABASES[‘default’].copy()
    db_dict[‘NAME’] = ‘{}_{}’.format(PROJECT_NAME, i)
    DATABASES[‘db_{}’.format(i)] = db_dict

    how to config the haystack and solr ? do you know? thanks.

  • Approve?

  • Hey Buddy

    being trying out ur tutorial and its greate i have a problem.

    i keep getting

    search/search.html (Template doesnt exist) any ti i run the django server runserver