Relevance judgments
Make quality explicit before comparing rankings.
Scale
Use a simple fixed rating scale and clear examples.
Context
Rate within market, assortment and available product data.
Multiple reviewers
Use additional assessment for subjective or important queries.
Version control
Update judgments when assortment or search intent demonstrably changes.