AIAIO: Our Blog

Construction of the Gatun lock, Panama Canal, 1912.

If you ask me how long it will take to do a familiar but substantial task, chances are I’ll give you a wrong answer. I will tell you, “That will take me two days,” when in fact it has never taken me two days. Perhaps it has always taken at least three, and usually four. But I’m unable to think accurately enough about the past to reach this conclusion, especially if you ask me directly. This is a rough explanation of the Planning Fallacy, one of the most fascinating and pervasive cognitive biases. We all suffer from it when we make estimates, and it is especially acute with off-the-cuff estimates intended for an audience. We feel the pressure of judgment on our estimates, and unconsciously seek approval by providing optimistic and incorrect numbers.

Hofstadter’s Law formulates the cognitive puzzle embedded in this fallacy:

It always takes longer than you expect, even when you take into account Hofstadter’s Law.

Knowing about the Planning Fallacy isn’t enough to escape its influence.

Multiply this estimation inaccuracy by a large number—a factor determined by the complexity of the project—and this accounts for the primary reason that most projects fail to deliver on time. The “padding” that individuals, their managers, and their managers’ managers regularly add to work estimates is often whittled away during the piecemeal negotiation that falls between the estimation and commitment phases, leaving the original, overly optimistic estimates.

The bottom line is, and always has been, that estimates we make at the beginning of a project, because they are made in ignorance of the future, and because the Planning Fallacy distorts our thinking, all too often range from Pollyannaish to tragically mistaken. See Jim Benson’s Why Plans Fail for a concise and revelatory spelunking into the depths of the planning mind.

When we allow a project to be bound by our initial estimates, whether they were constructed with the best of intentions but subject to the Planning Fallacy, hampered by a misunderstanding of the requirements, distorted to meet the demands of the client, or simply doomed to irrelevance by the inevitable array of exigencies that befall every project, we force reality into an inappropriate container.

But we still need to execute on the project plan, whatever its condition. Content strategy, UX, information architecture, conceptual design, applied design, rounds of approval and review: these initial phases may expand as they attempt to capture the entirety of emerging requirements, and subsequent phases become necessarily further compressed.

The technology team deals with this compressed time frame by (a) being galactic geniuses (I am not biased) and (b) making compromises. Compromises can be deliberative and explicit, panicked and hidden, or half-heartedly considered and partially documented by inline code comments.

Some compromises are driven by a clear-eyed assessment of the border between Minimally Sufficient and Fancy. Although programmers are famously lazy, they are just as often driven (by Larry Wall’s #3, Hubris) to engineer the perfect solution where an adequate one is the optimal path. Choosing adequate over perfect means that we can reserve precious engineering time for the hairier tasks, or the unexpected but inevitable road bumps that put further pressure on development schedules: bugs in the platform or a crucial module, foibles of the programming language, unexpected complexity in meeting a functional requirement, iterations between UI design and software implementation, and so on.

We can expect road bumps, but we can’t plan for them. If we start a development cycle with an unrealistic schedule estimate, then we can just expect to be late from the start. In an atmosphere like this, ill-considered compromises are almost inevitable. These compromises constitute the primary source of technical debt introduced to a new code base. We can only hope that the developers at least take the time to add TODO-style comments to mark the code they’d like to refactor in the future.

You can find these later:

$ find . -type f -exec grep -i todo {} \;

(Try that in Magento Enterprise 1.12’s app/code directory: 171 of them. It happens to the best of us.)

There are basically two strategies here:

1. Minimize bad technical debt* and stress by renegotiating the release schedule
2. Plan to address bad debt in a subsequent release

#1 is vastly preferable. #2 is only possible if you can isolate and track the debt, and push off developing new features in favor of addressing old problems. This often requires a heightened level of transparency between the technical and business teams. Otherwise, building new features on top of crufty, debt-ladened code will hinder progress on the features, slow your development velocity, increase the overall error rate, and reduce your ROI by alarming numbers.

How to minimize contact with this Planning-to-Debt problem: Plan to revise your estimates. Regardless of your project methodology and how it incorporates estimates, educate all project stakeholders as to the reality of estimation, if it’s not already abundantly clear. Help the team to understand that the process of estimation is more valuable than the estimates it produces.

* “Bad technical debt”: Not all technical debt is bad. Another useful distinction is short-term versus long-term debt.

In the latest releases of Magento Community and Enterprise Editions (1.7.0.2 and 1.12.0.2), there’s a core bug that throws a PHP Notice when you try to save certain configuration screens. For instance, when saving from the Advanced tab:

The code in question is this:

<?php

/**
 * Get field backend model
 */
$backendClass = $fieldConfig->backend_model;
if (!$backendClass) {
    $backendClass = 'core/config_data';
}

?>

The bug report on Magento is here (registration required).

It’s curious that both the bug report and this discussion suggest you add

<?php

unset($backendClass);

?>

before this code block. Strange, since this code block explicitly assigns new values to $backendClass one way or the other.

This solution has worked for us — just some additional value-checking:

<?php

/**
 * Get field backend model
 */
if (!is_object($fieldConfig) || empty($fieldConfig->backend_model)) {
    $backendClass = 'core/config_data';
}
else {
    $backendClass = $fieldConfig->backend_model;
}

?>

I’d be curious to hear any arguments about what memory-management magic the explicit unset() call would invoke here.

Gist file here: https://gist.github.com/4664829

Special thanks to Dan Pasette for his insights into how an engineering team like 10gen’s manages technical debt in their issue tracker.

If you work on software, you most likely use a bug tracker to track defects, manage development tasks, and plan releases. The collection of tickets, open and closed, is a data set that doubles as a repository of institutional memory about your software: what components have been the most complex (i.e., have spawned the most bugs)? Who worked on what, when? What issues did the developers grapple with over the course of delivering on a requirement or resolving a defect? What compromises did the team decide to make (or, worse, what were they forced to make) to hit a deadline? What’s never been fixed, left to fester in a dark corner?

You may have other documentation sources for your software: end-user docs, developer handoff docs, long and tortuous email chains, wiki pages, and inline documentation in the code base itself. But, if consistently used, a bug tracker can show the underbelly of the development process in remarkable detail. It is here where teams can probe their work history to uncover decisions and issues that we can categorize as technical debt. (Example: Open the longest-active project in your issue tracker, and find the oldest unresolved bug. Behold.)

And once we have classified bugs as technical debt, that can be its own component—a component we should consider as first-class, along with mission-critical and user-facing components. Unlike other components, which must be maintained and improved, technical debt must be managed. It is an emergent component that we should assume will always be present in a system.

This kind of emergent debt is what happens when we gain insight into the longstanding problems in a system, when we start to trace one problem back to a past decision, well-considered or otherwise. We can find the areas where we have built workarounds over the compromise and complexity we originally injected into the system.

And the status of these issues as bugs means that we have already filtered out most non-harmful technical debt (I am squarely in the “not all technical debt is bad” camp): if the team implemented a shortcut that never caused or registered as a problem, it’s not in the bug tracker to begin with.

Just as importantly, in some situations we can choose to split these bugs into actionable sub-tickets, and cast off the rest. If, for instance, an old bug has to do with a component that has architectural flaws, but which is used more and more rarely, we can choose just to address the surface problems, knowing that the underlying debt in this instance will get zeroed out when we finally sunset this component completely (this must be a real plan, of course, such as a replatforming project in the not-too-distant future). So we break out the surface problems into discrete bugs, and then we can close the hairy old bug and pat ourselves on the back.

For example: an e-commerce system has a year-old bug on the books concerning user feedback in the checkout flow. The underlying problem has to do with the poor integration of the payment gateway. You, the tech lead, are fresh from a meeting where the business owners agreed to a payment processor with a much cleaner integration. The transition is a few months away still, but all the bug-spawning cruft of the old integration is now scheduled to disappear. You can now split this bug into some constituent sub-bugs that make cosmetic changes sufficient to address the symptoms of the original bug report. You can also close out the year-old bug with references to the cosmetic bugs and the impending payment system change. Were a new payment system not in the works, this band-aid approach might seem an irresponsible way to manage the technical debt underlying the original bug; in the new reality, though, you’ve decided to acquire short-term debt that has a definite lifespan.

(Side note: this example highlights the breakdown in the strict financial debt analogy. Technical debt can, in some cases, just magically disappear. Don’t try this with your credit card.)

We can also use the bug tracker explicitly to expose and track technical debt as we take it on or discover it in a system. This is especially useful when a developer needs to surface and memorialize decisions, shortcuts, quandaries, and TODOs in order to implement a feature. Being able to enter bugs related to the development process, and to categorize them as debt and not feature bugs, allows the developer to stay focused on the implementation, while making the process more transparent and contributing to institutional memory.

I’ll explain by example. Let’s posit a developer: Cold cup of coffee perched dangerously on the edge of the desk, sitting back in an office chair, wearing overpriced headphones, perhaps sweating a bit, wrestling with the latest in a series of hairy performance problems near the end of a development cycle. The task at hand is to deal with a slow page load. In the process of tracing the execution chain, the developer runs into a run-on sentence of a method that cries out for refactoring. It’s also probable that some of this logic will be useful elsewhere, once it’s extracted from the monolithic method. It’s also clear, however, that this method is not impacting load time—for the purposes of the task at hand, it’s more of an itch to scratch.

Thanks to the GTD methodology, we know how to deal with these itches. The answer is not to lose focus on the task at hand, but rather to record the unrelated task and file it in a place where we can address it later. To do this, we need a simple filing method that allows us to categorize the new task efficiently, so that task creation is not itself a distracting activity.

Clearly, our developer needs to get this into the bug tracker. It takes thirty seconds to create a task called “Refactor Ai_Checkout_Helper::doThings().” The tech lead should provide a single, simple tag (or label, depending on your issue tracker’s terminology) for these debt-repaying tasks: shouldfix, refactor, or technicaldebt will work. The goal is to create a two-step method for recording technical debt-related bugs that developers find in the course of doing other work:

1. Enter minimal notes, including file and line reference.

2. Label as technical debt.

It would be fun to monitor activity logged against this label in the heat of a project (e.g., an RSS feed attached to the label), as you’d have an equivalent of the fantastic “WTFs per minute” quantification of technical debt.

For technologists who spend much of their time eyes-deep in the tools, platforms, and architectural drivers that great solutions require, it can be easy to feel isolated from the surrounding business. The business goals of the project and the financial context that surrounds and constrains it become, through the necessary processes of business analysis and project planning, several layers removed from the technology team’s internal representation of the project. Components to be built, system and application architectures, UML diagrams, task tickets, and burn down lists may be necessary constructs to run a development project, but they retain little to no understanding of the business context.

Just as the syntax of programming languages by and large lacks the ability to communicate the system architecture, the artifacts of project management and development planning lack the crucial ability to communicate the business context.

This is why the business and technology teams so often feel like separate factions, each harboring gripes about how the other lacks the context to understand the decisions that need to be made. (One beautiful and refreshing aspect of Ai’s culture is that it has the opposite character: we are pretty cozy here, despite our spacious office, and our size helps us to manage and minimize these divides.)

And here’s one big reason why “technical debt” has become a hot topic: it’s a concept that can do wonders to bridge the communication gap that often develops between technology and business teams over the course of a project.

This gap is itself a project risk, and as with all risks, we need to find the right tools to understand and mitigate it. Thus the (justified) popularity of “technical debt”.

But just as “technical debt” is a useful term for mitigating this risk, other terms can broaden or enforce the border around an IT team. One, a term more pervasive in larger organizations, is “the business”. This is how IT project managers often refer to that shadowy side of the organization that issues commands from its isolated realm, dictates that the IT group must translate into actions and project plans.

“The business” will place constraints of budget, schedule, platform on the project. (All too often, these get delivered to us via another unfortunate neologism: “the ask”. “Ask” is a verb. It just is. It’s a verb. When someone drops “the ask” on the meeting table, they’re presenting a hard object with no creator: “the ask” has arrived, ineluctable and undebatable.)

“The business” will make midstream decisions seemingly ignorant of implications to the project’s technological commitments. “The business” also, of course, writes the checks, so we feel we have no absolute leverage.

It is in dealing with these constraints from “the business” that we regularly incur technical debt: to meet a deadline, or to facilitate a sudden change in requirements, we commit to compromises in the code or architecture that we know we’ll need to fix someday. (“Someday”: as with financial debt, technical debt is a tool at your disposal; but you have to fix a date to this payment to keep your debt from ballooning.)

The key lesson here is that we can’t conceive of these decisions as technology decisions. As Steve McConnell notes, “At the end of the day, all [technical] decisions made in this context are business decisions.” The business and technology teams are partners in the success of a project. Business decisions must take technology into account, and vice versa.

We can take it a step further, in fact: there is no “the business”. Each one of us is The Business.

Considering ourselves, the technologists, to be The Business means internalizing that each line of code we write, each component we build, each compromise we make affects the business context of the project, and ultimately the success of the wider organization. What’s the business value of documenting this code? What’s the business value of building this test script? There’s no reason why QA engineers and developers should labor in the absence of this notion of business value. Knowing the business context helps us make intelligent decisions, spend our time and energy wisely to focus on value, not problems or minutiae.

If we erase this construct of a separate “business”, we give the project a huge leg up: now, project direction needs to include business and technological context. Now, we are forced to have cross-disciplinary conversation around difficult decisions. Now, when a high-value and very difficult requirement becomes an architectural driver for the technology team, we can understand and plan around this big hurdle in the context of its overall importance.

The interest in recent years in managing technical debt is just this: an increasing interest in fostering common understanding around difficult technical decisions, and in providing institutional memory of debt incurred so that the organization can agree, and remember, to pay down that debt in the future.

I recently needed a way to add SSL to varnish and decided to give Pound a try. There are some great howtos available on the web, but there is one thing I don’t like about the suggested configurations. The general suggestion is to add this to your pound config

HeadRemove "X-Forwarded-Proto"
AddHeader "X-Forwarded-Proto: https"

and then to check for that header in your application, varnish, or whereever you need to check for SSL. However, by manually sending an “X-Forwarded-Proto: https” header directly to varnish on port 80, you can trick your backend application into thinking you are requesting information over HTTPS when you aren’t. While I don’t think this is exploitable by itself, I certainly don’t want to leave any room for hacker mischief.

My suggestion is to add one additional secret header in your pound config, and then sanitize the headers in varnish if that secret header is missing. For example, in my pound config, I added this:

HeadRemove "X-Forwarded-Proto"
AddHeader "X-Forwarded-Proto: https"
AddHeader "X-Pound: PUTARANDOMSTRINGHERE"

And in my varnish vcl_recv:

if (req.http.X-Pound == "PUTARANDOMSTRINGHERE" && req.http.X-Forwarded-Proto == "https") {
    unset req.http.X-Pound;
    #take any extra needed actions for SSL here
} else {
    unset req.http.X-Pound;
    unset req.http.X-Forwarded-Proto;
}

Now when I check for the X-Forwarded-Proto header in my application, I can be sure that the client really is making the request over HTTPS. Notice that I always remove the X-Pound header after I’ve checked for it, even if it is valid. There is no need for the application to ever see that header – no need to risk any potential leakage of my secret header, which could potentially happen if a debug setting is ever left on in the application.

A headless Selenium testing system is an ideal addition to any development workflow. Selenium reduces testing time, and integrates into CI tools such as Jenkins. The benefits are great, but only *if* you can get your tests to run. One of the most time consuming problems that can arise from a headless system is a failed test. How can you debug, when you can’t see the browser? Ideally, you will test your scripts locally before running them on the headless system, but anyone who has even dabbled in the world of automation knows that a single procedure can yield very different results on different systems and in different environments. You can deal with the differences, but first you have to see them.  After much searching, I could not find a straight answer as to how to export this display to my Windows machine from our Linux server. It is really quite simple, and can be done in a few steps.  In order to display a headless Selenium test on a Windows machine from a Linux server, you must first be able to have an X window (X11) server running on your Windows machine. This is the underlying window display environment common to Unix systems. I used Xming, as it is the easiest to use and involves almost no setup.

You can download Xming from here.

When the installer finishes, run the Xming server. This doesn’t do anything that you can see, but starts the X11 environment on top of Windows.  Now you need PuTTY, the SSH client.

You can download PuTTY here.

Now that you have installed PuTTY, run the executable. In the left hand category listing for the settings, click on the X11 tab and enable X11 forwarding.

Type in the host name or IP of the remote machine and connect. Type “firefox” in the command line of your PuTTY shell (this assumes you have Firefox on your system).  This should pop the Firefox browser!  Now that you can get the remote Firefox instance to show on your Windows machine, you can run Selenium through PuTTY and debug on the fly with the command:

java -jar (PATH-TO)/selenium-server.jar -trustAllSSLCertificates \
        -htmlSuite BROWSER URL  (PATH-TO)/SUITE  (PATH-TO)/LOG

I’m adding drag/drop uploading to the django admin for one of our open source projects called Stager. A blog post about that will follow, it’s not screen-shot ready yet. While doing this I knew we needed a pretty seamless transition after the upload finished, and that we would have to refresh the inline. I didn’t want a full page refresh, so let’s ajax it in.

For these examples just assume that we have a parent CompAdmin which has an model of Comp and an inline called CompSlideInline. We store the instance of the Comp in comp.

from django.template import loader, Context
from django.contrib.admin import helpers
from django.db import transaction
from django.contrib import admin

comp = Comp.objects.get(id=comp_id)
#get the current site
admin_site = admin.site
compAdmin = CompAdmin(Comp, admin_site)

#get all possible inlines for the parent Admin
inline_instances = compAdmin.get_inline_instances(request)
prefixes = {}

for FormSet, inline in zip(compAdmin.get_formsets(request, comp), inline_instances):
    #get the inline of interest and generate it's formset
    if isinstance(inline, CompSlideInline):
        prefix = FormSet.get_default_prefix()
        prefixes[prefix] = prefixes.get(prefix, 0) + 1
        if prefixes[prefix] != 1 or not prefix:
            prefix = "%s-%s" % (prefix, prefixes[prefix])
        formset = FormSet(instance=comp, prefix=prefix, queryset=inline.queryset(request))

#get possible fieldsets, readonly, and prepopulated information for the parent Admin
fieldsets = list(inline.get_fieldsets(request, comp))
readonly = list(inline.get_readonly_fields(request, comp))
prepopulated = dict(inline.get_prepopulated_fields(request, comp))

#generate the inline formset
inline_admin_formset = helpers.InlineAdminFormSet(inline, formset,
            fieldsets, prepopulated, readonly, model_admin=compAdmin)

#render the template
t = loader.get_template('admin/staging/edit_inline/_comp_slide_drag_upload_ajax.html')
c = Context({ 'inline_admin_formset': inline_admin_formset })
rendered = t.render(c)

Inevitably, when you get into the weeds with a Magento build, you’ll need to run some big, hairy process — perhaps even on a regular basis. Perhaps it involves processing images, or updating data, or creating reports. Perhaps it runs on cron, but you’d also like to run it from the command line. And if it’s destructive — i.e., it alters data — you definitely want to make sure you’re only running one at a time.

Magento has a nice semaphore locking process handler built in to its indexers. In order for a reindexing process to run, the process needs to obtain a lock. The code lives in the Mage_Index_Model_Process class, and has helpful methods such as isLocked(), lockAndBlock(), and unlock(). These look for and manipulate files in your application’s var/locks directory.

The implementation is a tried and true locking mechanism that you could write yourself, but why bloat an already massive code base? We can just repurpose this relatively self-contained functionality whenever we need to lock a process.

Solr is a very powerful search tool and it is pretty easy to get the basics, such as full text search, facets, and related assets up and running pretty quickly. We will be using haystack to do the communication between Django and Solr. All code for this can be viewed on github.

Install

Assuming you already have Django up and running, the first thing we need to do is install Solr.

curl -O http://mirrors.gigenet.com/apache/lucene/solr/4.0.0-BETA/apache-solr-4.0.0-BETA.zip
unzip apache-solr-4.0.0-BETA.zip
cd apache-solr-4.0.0-BETA
cd example
java -jar start.jar

Next install pysolr and haystack. (At the time of this writing the git checkout of haystack works better with the Solr 4.0 beta then the 1.2.7 that’s in pip.)

pip install pysolr
pip install -e https://github.com/toastdriven/django-haystack.git

Add ‘haystack’ to INSTALLED_APPS in settings.py and add the following haystack connection:

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.solr_backend.SolrEngine',
        'URL': 'http://127.0.0.1:8983/solr'
    },
}

Full Text Search

For the example, we’re going to create a simple job database that a recruiter might use. Here is the model:

from django.db import models
from django.contrib.localflavor.us import models as us_models

JOB_TYPES = (
    ('pt', 'Part Time'),
    ('ft', 'Full Time'),
    ('ct', 'Contract')
)

class Company(models.Model):
    name = models.CharField(max_length=64)
    address = models.TextField(blank=True, null=True)
    contact_email = models.EmailField()

    def __unicode__(self):
        return self.name

class Location(models.Model):
    city = models.CharField(max_length=64)
    state = us_models.USStateField()

    def __unicode__(self):
        return "%s, %s" % (self.city, self.state)

class Job(models.Model):
    name = models.CharField(max_length=64)
    description = models.TextField()
    salary = models.CharField(max_length=64, blank=True, null=True)
    type = models.CharField(max_length=2, choices=JOB_TYPES)
    company = models.ForeignKey(Company, related_name='jobs')
    location = models.ForeignKey(Location, related_name='location_jobs')
    contact_email = models.EmailField(blank=True, null=True)
    added_at = models.DateTimeField(auto_now=True)

    def __unicode__(self):
        return self.name

    def get_contact_email(self):
        if self.contact_email:
            return self.contact_email
        return self.company.contact_email

The next step is to create the SearchIndex object that will be used to transpose to data to Solr. save this as search_indexes.py in the same folder as your models.py. The text field with its template will be used for full text search on Solr. The other two fields will be used to faceted (drill down) navigation. For more details on this file, check out the haystack tutorial.

class JobIndex(indexes.SearchIndex, indexes.Indexable):
    text = indexes.CharField(document=True, use_template=True)
    type = indexes.CharField(model_attr='type', faceted=True)
    location = indexes.CharField(model_attr='location', faceted=True)

    def get_model(self):
        return Job

    def index_queryset(self):
        return self.get_model().objects.all()

Create the search index template in your template folder with the following naming convention: search/indexes/[app]/[model]_text.txt
For us, this is templates/search/indexes/jobs/job_text.txt

{{ object.name }}
{{ object.description }}
{{ object.salary }}
{{ object.type }}
{{ object.added_at }}

Now, lets get our data into Solr. Run ./manage.py build_solr_schema to generate a schema.xml file. Move this into example\solr\conf in your Solr install. Note: if using Solr 4, edit this file and replace stopwords_en.txt with lang/stopwords_en.txt in all locations. To test everything and load your data, run: manage.py rebuild_index Subsequent updates can be made with: manage.py update_index.

If that all worked we can start working on the front-end to see the data in Django. Add this to your urls.py

(r'^$', include('haystack.urls')),

At this point there are at least two templates we’ll need. One for the search results page, and a sub-template to represent each item we are pulling back. My example uses twitter bootstrap for some layout help and styling, see my base.html here if interested.

Create templates/search/search.html
This gives you a basic search form, the results, and pagination for a number of results

{% extends 'base.html' %}

{% block hero_text %}Search{% endblock %}
{% block header %}
Click around!

{% endblock %}

{% block content %}</pre>
<div class="span12">
<h1>Search</h1>
<form class=".form-search" action="." method="get">{{ form.as_table }}
 <input type="submit" value="Search" /></form></div>
<pre>
{% if query %}</pre>
<div class="span8">
<h3>Results</h3>
<div id="accordion2" class="accordion">{% for result in page.object_list %}
 {% include 'search/_result_object.html' %}
 {% empty %}

No results found.

 {% endfor %}</div>
 {% if page.has_previous or page.has_next %}
<div>{% if page.has_previous %}<a href="?q={{ query }}&page={{ page.previous_page_number }}">{% endif %}« Previous{% if page.has_previous %}</a>{% endif %}
 |
 {% if page.has_next %}<a href="?q={{ query }}&page={{ page.next_page_number }}">{% endif %}Next »{% if page.has_next %}</a>{% endif %}</div>
 {% endif %}</div>
<pre>
{% else %}

{% endif %}
{% endblock %}

And the templates/search/_result_object.txt

{% with obj=result.object %}</pre>
<div class="accordion-group">
<div class="accordion-heading"><a class="accordion-toggle" href="#collapse_{{ obj.id }}" data-toggle="collapse" data-parent="#accordion2">
 {{ obj.name }}
 </a>
<div style="padding: 8px 15px;">
Company: {{ obj.company }}

Type: {{ obj.type }}

 {% if obj.salary %}
Salary: {{ obj.salary }}

{% endif %}

Location: {{ obj.location }}</div>
</div>
<div id="collapse_{{ obj.id }}" class="accordion-body collapse in">
<div class="accordion-inner">
Contact: <a href="mailto:{{ obj.get_contact_email }}">{{ obj.get_contact_email }}</a>

 {{ obj.description }}</div>
</div>
</div>
<pre>
{% endwith %}

Start up your dev server for search!

Related Items

Adding Related Items is as simple as using the related_content tag in the haystack more_like_this tag library and tweaking out Solr config. Open up solrconfig.xml and add a MoreLikeThisHandler within the tag:

<requestHandler name="/mlt" class="solr.MoreLikeThisHandler" />

Our full _result_object.html now looks like this:

{% load more_like_this %}

{% with obj=result.object %}
<div class="accordion-group">
    <div class="accordion-heading">
        <a class="accordion-toggle" data-toggle="collapse" data-parent="#accordion2" href="#collapse_{{ obj.id }}">
            {{ obj.name }}
        </a>
        <div style="padding: 8px 15px;">
            <p>Company: {{ obj.company }}</p>
            <p>Type: {{ obj.type }}</p>
            {% if obj.salary %}<p>Salary: {{ obj.salary }}</p>{% endif %}
            <p>Location: {{ obj.location }}</p>
        </div>
    </div>
    <div id="collapse_{{ obj.id }}" class="accordion-body collapse in">
        <div class="accordion-inner">
            <p>Contact: <a href="mailto:{{ obj.get_contact_email }}">{{ obj.get_contact_email }}</a></p>
            {{ obj.description }}
            {% more_like_this obj as related_content limit 5  %}
            {% if related_content %}
                <div>
                    <br>
                    <p><strong>Related:</strong></p>
                    <ul>
                        {% for related in related_content %}
                            <li><a>{{ related.object.name }}</a></li>
                        {% endfor %}
                    </ul>
                </div>
            {% endif %}
        </div>
    </div>
</div>
{% endwith %}

Facets

To get our type and location facets, we’ll have to add them to a queryset and pass this to a FacetedSearchView instead of the default one. urls.py now looks like this:

from django.conf.urls import patterns, include, url
from haystack.forms import FacetedSearchForm
from haystack.query import SearchQuerySet
from haystack.views import FacetedSearchView

sqs = SearchQuerySet().facet('type').facet('location')

urlpatterns = patterns('haystack.views',
    url(r'^$', FacetedSearchView(form_class=FacetedSearchForm, searchqueryset=sqs), name='haystack_search'),
)

Then, we can use the generated facets in the search template in the facets variable

{% extends 'base.html' %}

{% block hero_text %}Search{% endblock %}
{% block header %}<p>Click around!</p>{% endblock %}

{% block content %}
<div class="span12">
    <h1>Search</h1>
    <form method="get" action="." class=".form-search">
        <table>
            {{ form.as_table }}
        </table>
        <input type="submit" value="Search">
    </form>
</div>
        {% if query %}
            <div class="span2">
                <h3>Filter</h3>
                {% if facets.fields.type %}
                    <div>
                        <h4>Type</h4>
                        <ul>
                        {% for type in facets.fields.type %}
                            <li><a href="{{ request.get_full_path }}&amp;selected_facets=type_exact:{{ type.0|urlencode }}">{{ type.0 }}</a> ({{ type.1 }})</li>
                        {% endfor %}
                        </ul>
                    </div>
                {% endif %}
                {% if facets.fields.location %}
                    <div>
                        <h4>Location</h4>
                        <ul>
                        {% for location in facets.fields.location %}
                            <li><a href="{{ request.get_full_path }}&amp;selected_facets=location_exact:{{ location.0|urlencode }}">{{ location.0 }}</a> ({{ location.1 }})</li>
                        {% endfor %}
                        </ul>
                    </div>
                {% endif %}
            </div>
            <div class="span6">
                <h3>Results</h3>
                <div class="accordion" id="accordion2">
                    {% for result in page.object_list %}
                        {% include 'search/_result_object.html' %}
                    {% empty %}
                        <p>No results found.</p>
                    {% endfor %}
                </div>

                {% if page.has_previous or page.has_next %}
                    <div>
                        {% if page.has_previous %}<a href="?q={{ query }}&amp;page={{ page.previous_page_number }}">{% endif %}&laquo; Previous{% if page.has_previous %}</a>{% endif %}
                        |
                        {% if page.has_next %}<a href="?q={{ query }}&amp;page={{ page.next_page_number }}">{% endif %}Next &raquo;{% if page.has_next %}</a>{% endif %}
                    </div>
                {% endif %}
            </div>
        {% else %}
            <div class="span6">
                {# Show some example queries to run, maybe query syntax, something else? #}
            </div>
        {% endif %}
{% endblock %}

And we’re done! As I said, check out the haystack documentation for more information. Leave any questions in the comments and I’ll be sure to answer them. Spelling suggestions to come in the next post.

While doing routine sanity checks, one of our QA Engineers, Sammy Shaar, was alerted about an important Magento security update. The vulnerability potentially allows an attacker to read any file on the web server where the Zend XMLRPC functionality is enabled. This might include password files, configuration files, and possibly even databases if they are stored on the same machine as the Magento web server.

To see if you site has been affected, please see this page.

Luckily, Magento has released patches for all supported versions:

• Magento Enterprise Edition and Professional Edition merchants:
  You may access the Zend Security Upgrade patch from Patches & Support for your product in the Downloads section of your Magento account. Account log-in is required.
  Download

• Magento Community Edition merchants:
  Community Edition 1.4.0.0 through 1.4.1.1
  Community Edition 1.4.2.0
  Community Edition 1.5.0.0 through 1.7.0.1

To install the patch, place the patch file in the root of your Magento site and run the following command:

patch -p0 < zendxml_fix.patch

If you don’t have ssh access or patch installed on your machine, please see this stack overflow post for alternative methods.

Please Note: All current Ai and Canopy sites have been patched