Querying Vespa¶

This guide goes through how to query a Vespa instance using the Query API and https://cord19.vespa.ai/ and https://search.vespa.ai/ apps as examples.

Refer to troubleshooting for any problem when running this guide.

You can run this tutorial in Google Colab:

In [1]:

!pip3 install pyvespa

!pip3 install pyvespa

Connect to a running Vespa instance.

In [2]:

from vespa.application import Vespa
from vespa.io import VespaQueryResponse
from vespa.exceptions import VespaError

app = Vespa(url="https://api.cord19.vespa.ai")

from vespa.application import Vespa from vespa.io import VespaQueryResponse from vespa.exceptions import VespaError app = Vespa(url="https://api.cord19.vespa.ai")

See the Vespa query language for Vespa query api request parameters.

The YQL userQuery() operator uses the query read from query. The query also specifies to use the app-specific bm25 rank profile. The code uses context manager with session statement to make sure that connection pools are released. If you attempt to make multiple queries, this is important as each query will not have to set up new connections.

In [3]:

with app.syncio() as session:
    response: VespaQueryResponse = session.query(
        yql="select documentid, cord_uid, title, abstract from sources * where userQuery()",
        hits=1,
        query="Is remdesivir an effective treatment for COVID-19?",
        ranking="bm25",
    )
    print(response.is_successful())
    print(response.url)

with app.syncio() as session: response: VespaQueryResponse = session.query( yql="select documentid, cord_uid, title, abstract from sources * where userQuery()", hits=1, query="Is remdesivir an effective treatment for COVID-19?", ranking="bm25", ) print(response.is_successful()) print(response.url)

True
https://api.cord19.vespa.ai/search/?yql=select+documentid%2C+cord_uid%2C+title%2C+abstract+from+sources+%2A+where+userQuery%28%29&hits=1&query=Is+remdesivir+an+effective+treatment+for+COVID-19%3F&ranking=bm25

Alternatively, if the native Vespa query parameter contains ".", which cannot be used as a kwarg, the parameters can be sent as HTTP POST with the body argument. In this case, ranking is an alias of ranking.profile, but using ranking.profile as a **kwargs argument is not allowed in python. This will combine HTTP parameters with an HTTP POST body.

In [4]:

with app.syncio() as session:
    response: VespaQueryResponse = session.query(
        hits=1,
        body={
            "yql": "select documentid, cord_uid, title, abstract from sources * where userQuery()",
            "query": "Is remdesivir an effective treatment for COVID-19?",
            "ranking.profile": "bm25",
            "presentation.timing": True,
        },
    )
    print(response.is_successful())

with app.syncio() as session: response: VespaQueryResponse = session.query( hits=1, body={ "yql": "select documentid, cord_uid, title, abstract from sources * where userQuery()", "query": "Is remdesivir an effective treatment for COVID-19?", "ranking.profile": "bm25", "presentation.timing": True, }, ) print(response.is_successful())

True

The query specified that we wanted one hit:

In [5]:

response.hits

response.hits

Out[5]:

[{'id': 'id:covid-19:doc::534720',
  'relevance': 26.6769101612402,
  'source': 'content',
  'fields': {'title': 'A Review on <hi>Remdesivir</hi>: A Possible Promising Agent for the <hi>Treatment</hi> of <hi>COVID</hi>-<hi>19</hi>',
   'abstract': '<sep />manufacturing of specific therapeutics and vaccines to treat <hi>COVID</hi>-<hi>19</hi> are time-consuming processes. At this time, using available conventional therapeutics along with other <hi>treatment</hi> options may be useful to fight <hi>COVID</hi>-<hi>19</hi>. In different clinical trials, efficacy of <hi>remdesivir</hi> (GS-5734) against Ebola virus has been demonstrated. Moreover, <hi>remdesivir</hi> may be an <hi>effective</hi> therapy in vitro and in animal models infected by SARS and MERS coronaviruses. Hence, the drug may be theoretically <hi>effective</hi> against SARS-CoV-2. <hi>Remdesivir</hi><sep />',
   'documentid': 'id:covid-19:doc::534720',
   'cord_uid': 'xej338lo'}}]

Example of iterating over the returned hits obtained from response.hits, extracting the cord_uid field:

In [6]:

[hit["fields"]["cord_uid"] for hit in response.hits]

[hit["fields"]["cord_uid"] for hit in response.hits]

Out[6]:

['xej338lo']

Access the full JSON response in the Vespa default JSON result format:

In [7]:

response.json

response.json

Out[7]:

{'timing': {'querytime': 0.007, 'summaryfetchtime': 0.0, 'searchtime': 0.008},
 'root': {'id': 'toplevel',
  'relevance': 1.0,
  'fields': {'totalCount': 2373},
  'coverage': {'coverage': 100,
   'documents': 976355,
   'full': True,
   'nodes': 2,
   'results': 1,
   'resultsFull': 1},
  'children': [{'id': 'id:covid-19:doc::534720',
    'relevance': 26.6769101612402,
    'source': 'content',
    'fields': {'title': 'A Review on <hi>Remdesivir</hi>: A Possible Promising Agent for the <hi>Treatment</hi> of <hi>COVID</hi>-<hi>19</hi>',
     'abstract': '<sep />manufacturing of specific therapeutics and vaccines to treat <hi>COVID</hi>-<hi>19</hi> are time-consuming processes. At this time, using available conventional therapeutics along with other <hi>treatment</hi> options may be useful to fight <hi>COVID</hi>-<hi>19</hi>. In different clinical trials, efficacy of <hi>remdesivir</hi> (GS-5734) against Ebola virus has been demonstrated. Moreover, <hi>remdesivir</hi> may be an <hi>effective</hi> therapy in vitro and in animal models infected by SARS and MERS coronaviruses. Hence, the drug may be theoretically <hi>effective</hi> against SARS-CoV-2. <hi>Remdesivir</hi><sep />',
     'documentid': 'id:covid-19:doc::534720',
     'cord_uid': 'xej338lo'}}]}}

Query Performance¶

There are several things that impact end-to-end query performance:

• HTTP layer performance, connecting handling, mututal TLS handshake and network round-trip latency
  • Make sure to re-use connections using context manager with vespa.app.syncio(): to avoid setting up new connections for every unique query. See http best practises
  • The size of the fields and the number of hits requested also greatly impact network performance; a larger payload means higher latency.
  • By adding "presentation.timing": True as a request parameter, the Vespa response includes the server-side processing (also including reading the query from the network, but not delivering the result over the network). This can be handy for debugging latency.
• Vespa performance, the features used inside the Vespa instance.

In [8]:

with app.syncio(connections=12) as session:
    response: VespaQueryResponse = session.query(
        hits=1,
        body={
            "yql": "select documentid, cord_uid, title, abstract from sources * where userQuery()",
            "query": "Is remdesivir an effective treatment for COVID-19?",
            "ranking.profile": "bm25",
            "presentation.timing": True,
        },
    )
    print(response.is_successful())

with app.syncio(connections=12) as session: response: VespaQueryResponse = session.query( hits=1, body={ "yql": "select documentid, cord_uid, title, abstract from sources * where userQuery()", "query": "Is remdesivir an effective treatment for COVID-19?", "ranking.profile": "bm25", "presentation.timing": True, }, ) print(response.is_successful())

True

Compressing queries¶

The VespaSync class has a compress argument that can be used to compress the query before sending it to Vespa. This can be useful when the query is large and/or the network is slow. The compression is done using gzip, and is supported by Vespa.

By default, the compress argument is set to "auto", which means that the query will be compressed if the size of the query is larger than 1024 bytes. The compress argument can also be set to True or False to force the query to be compressed or not, respectively.

The compression will be applied to both queries and feed operations. (HTTP POST or PUT requests).

In [9]:

import time

# Will not compress the request, as body is less than 1024 bytes
with app.syncio(connections=1, compress="auto") as session:
    response: VespaQueryResponse = session.query(
        hits=1,
        body={
            "yql": "select documentid, cord_uid, title, abstract from sources * where userQuery()",
            "query": "Is remdesivir an effective treatment for COVID-19?",
            "ranking.profile": "bm25",
            "presentation.timing": True,
        },
    )
    print(response.is_successful())

# Will compress, as the size of the body exceeds 1024 bytes.
large_body = {
    "yql": "select documentid, cord_uid, title, abstract from sources * where userQuery()",
    "query": "Is remdesivir an effective treatment for COVID-19?",
    "input.query(q)": "asdf" * 10000,
    "ranking.profile": "bm25",
    "presentation.timing": True,
}
compress_time = {}

with app.syncio(connections=1, compress=True) as session:
    start_time = time.time()
    response: VespaQueryResponse = session.query(
        hits=1,
        body=large_body,
    )
    end_time = time.time()
    compress_time["force_compression"] = end_time - start_time
    print(response.is_successful())

with app.syncio(connections=1, compress="auto") as session:
    start_time = time.time()
    response: VespaQueryResponse = session.query(
        hits=1,
        body=large_body,
    )
    end_time = time.time()
    compress_time["auto"] = end_time - start_time
    print(response.is_successful())

# Force no compression
with app.syncio(compress=False) as session:
    start_time = time.time()
    response: VespaQueryResponse = session.query(
        hits=1,
        body=large_body,
        timeout="5s",
    )
    end_time = time.time()
    compress_time["no_compression"] = end_time - start_time
    print(response.is_successful())

import time # Will not compress the request, as body is less than 1024 bytes with app.syncio(connections=1, compress="auto") as session: response: VespaQueryResponse = session.query( hits=1, body={ "yql": "select documentid, cord_uid, title, abstract from sources * where userQuery()", "query": "Is remdesivir an effective treatment for COVID-19?", "ranking.profile": "bm25", "presentation.timing": True, }, ) print(response.is_successful()) # Will compress, as the size of the body exceeds 1024 bytes. large_body = { "yql": "select documentid, cord_uid, title, abstract from sources * where userQuery()", "query": "Is remdesivir an effective treatment for COVID-19?", "input.query(q)": "asdf" * 10000, "ranking.profile": "bm25", "presentation.timing": True, } compress_time = {} with app.syncio(connections=1, compress=True) as session: start_time = time.time() response: VespaQueryResponse = session.query( hits=1, body=large_body, ) end_time = time.time() compress_time["force_compression"] = end_time - start_time print(response.is_successful()) with app.syncio(connections=1, compress="auto") as session: start_time = time.time() response: VespaQueryResponse = session.query( hits=1, body=large_body, ) end_time = time.time() compress_time["auto"] = end_time - start_time print(response.is_successful()) # Force no compression with app.syncio(compress=False) as session: start_time = time.time() response: VespaQueryResponse = session.query( hits=1, body=large_body, timeout="5s", ) end_time = time.time() compress_time["no_compression"] = end_time - start_time print(response.is_successful())

True
True
True
True

In [10]:

compress_time

compress_time

Out[10]:

{'force_compression': 0.6639358997344971,
 'auto': 0.6602010726928711,
 'no_compression': 1.3003361225128174}

The differences will be more significant the larger the size of the body, and the slower the network. It might be beneficial to perform a proper benchmarking if performance is critical for your application.

Running Queries asynchronously¶

If you want to benchmark the capacity of a Vespa application, we suggest using vespa-fbench, which is a load generator tool that lets you measure throughput and latency with a predefined number of clients. Vespa-fbench is not Vespa-specific, and can be used to benchmark any HTTP service.

Another option is to use the Open Source k6 load testing tool.

If you want to run multiple queries from pyvespa, we suggest using the async client. Below, we will demonstrate a simple example of running 100 queries in parallel using the async client, and capture both the server-reported times and the client-reported times (including network latency).

In [11]:

# This cell is necessary when running async code in Jupyter Notebooks, as it already runs an event loop
import nest_asyncio

nest_asyncio.apply()

# This cell is necessary when running async code in Jupyter Notebooks, as it already runs an event loop import nest_asyncio nest_asyncio.apply()

In [12]:

import asyncio
import time


# Define a single query function that takes a session
async def run_query_async(session, body):
    start_time = time.time()
    response = await session.query(body=body)
    end_time = time.time()
    return response, end_time - start_time


query = {
    "yql": "select documentid, cord_uid, title, abstract from sources * where userQuery()",
    "query": "Is remdesivir an effective treatment for COVID-19?",
    "ranking.profile": "bm25",
    "presentation.timing": True,
}

# List of queries with hits from 1 to 100
queries = [{**query, "hits": hits} for hits in range(1, 101)]


# Define a function to run multiple queries concurrently using the same session
async def run_multiple_queries(queries):
    # Async client uses HTTP/2, so we only need one connection
    async with app.asyncio(connections=1) as session:  # Reuse the same session
        tasks = []
        for q in queries:
            tasks.append(run_query_async(session, q))
        responses = await asyncio.gather(*tasks)
    return responses


# Run the queries concurrently
start_time = time.time()
responses = asyncio.run(run_multiple_queries(queries))
end_time = time.time()
print(f"Total time: {end_time - start_time:.2f} seconds")
# Print QPS
print(f"QPS: {len(queries) / (end_time - start_time):.2f}")

import asyncio import time # Define a single query function that takes a session async def run_query_async(session, body): start_time = time.time() response = await session.query(body=body) end_time = time.time() return response, end_time - start_time query = { "yql": "select documentid, cord_uid, title, abstract from sources * where userQuery()", "query": "Is remdesivir an effective treatment for COVID-19?", "ranking.profile": "bm25", "presentation.timing": True, } # List of queries with hits from 1 to 100 queries = [{**query, "hits": hits} for hits in range(1, 101)] # Define a function to run multiple queries concurrently using the same session async def run_multiple_queries(queries): # Async client uses HTTP/2, so we only need one connection async with app.asyncio(connections=1) as session: # Reuse the same session tasks = [] for q in queries: tasks.append(run_query_async(session, q)) responses = await asyncio.gather(*tasks) return responses # Run the queries concurrently start_time = time.time() responses = asyncio.run(run_multiple_queries(queries)) end_time = time.time() print(f"Total time: {end_time - start_time:.2f} seconds") # Print QPS print(f"QPS: {len(queries) / (end_time - start_time):.2f}")

Total time: 10.17 seconds
QPS: 9.84

In [13]:

dict_responses = [response.json | {"time": timing} for response, timing in responses]

dict_responses = [response.json | {"time": timing} for response, timing in responses]

In [14]:

dict_responses[0]

dict_responses[0]

Out[14]:

{'timing': {'querytime': 0.005, 'summaryfetchtime': 0.0, 'searchtime': 0.006},
 'root': {'id': 'toplevel',
  'relevance': 1.0,
  'fields': {'totalCount': 2395},
  'coverage': {'coverage': 100,
   'documents': 976355,
   'full': True,
   'nodes': 2,
   'results': 1,
   'resultsFull': 1},
  'children': [{'id': 'id:covid-19:doc::534720',
    'relevance': 26.6769101612402,
    'source': 'content',
    'fields': {'title': 'A Review on <hi>Remdesivir</hi>: A Possible Promising Agent for the <hi>Treatment</hi> of <hi>COVID</hi>-<hi>19</hi>',
     'abstract': '<sep />manufacturing of specific therapeutics and vaccines to treat <hi>COVID</hi>-<hi>19</hi> are time-consuming processes. At this time, using available conventional therapeutics along with other <hi>treatment</hi> options may be useful to fight <hi>COVID</hi>-<hi>19</hi>. In different clinical trials, efficacy of <hi>remdesivir</hi> (GS-5734) against Ebola virus has been demonstrated. Moreover, <hi>remdesivir</hi> may be an <hi>effective</hi> therapy in vitro and in animal models infected by SARS and MERS coronaviruses. Hence, the drug may be theoretically <hi>effective</hi> against SARS-CoV-2. <hi>Remdesivir</hi><sep />',
     'documentid': 'id:covid-19:doc::534720',
     'cord_uid': 'xej338lo'}}]},
 'time': 0.6231591701507568}

In [15]:

# Create a pandas DataFrame with the responses
import pandas as pd

df = pd.DataFrame(
    [
        {
            "hits": len(response["root"]["children"]),
            "search_time": response["timing"]["searchtime"],
            "query_time": response["timing"]["querytime"],
            "summary_time": response["timing"]["summaryfetchtime"],
            "total_time": response["time"],
        }
        for response in dict_responses
    ]
)
df

# Create a pandas DataFrame with the responses import pandas as pd df = pd.DataFrame( [ { "hits": len(response["root"]["children"]), "search_time": response["timing"]["searchtime"], "query_time": response["timing"]["querytime"], "summary_time": response["timing"]["summaryfetchtime"], "total_time": response["time"], } for response in dict_responses ] ) df

Out[15]:

	hits	search_time	query_time	summary_time	total_time
0	1	0.006	0.005	0.000	0.623159
1	2	0.014	0.010	0.003	6.300802
2	3	0.012	0.010	0.001	6.300616
3	4	0.012	0.010	0.001	2.144979
4	5	0.009	0.007	0.001	4.597889
...	...	...	...	...	...
95	96	0.066	0.031	0.034	10.134708
96	97	0.058	0.030	0.027	10.148543
97	98	0.029	0.011	0.017	10.148778
98	99	0.051	0.018	0.032	10.135024
99	100	0.043	0.015	0.027	10.149029

100 rows × 5 columns

Error handling¶

Vespa's default query timeout is 500ms; Pyvespa will by default retry up to 3 times for queries that return response codes like 429, 500,503 and 504. A VespaError is raised if retries did not end up with success. In the following example, we set a very low timeout of 1ms which will cause Vespa to time out the request, and it returns a 504 http error code. The underlying error is wrapped in a VespaError with the payload error message returned from Vespa:

In [16]:

with app.syncio(connections=12) as session:
    try:
        response: VespaQueryResponse = session.query(
            hits=1,
            body={
                "yql": "select * from sources * where userQuery()",
                "query": "Is remdesivir an effective treatment for COVID-19?",
                "timeout": "1ms",
            },
        )
        print(response.is_successful())
    except VespaError as e:
        print(str(e))

with app.syncio(connections=12) as session: try: response: VespaQueryResponse = session.query( hits=1, body={ "yql": "select * from sources * where userQuery()", "query": "Is remdesivir an effective treatment for COVID-19?", "timeout": "1ms", }, ) print(response.is_successful()) except VespaError as e: print(str(e))

[{'code': 12, 'summary': 'Timed out', 'message': 'No time left after waiting for 1ms to execute query'}]

In the following example, we forgot to include the query parameter but still reference it in the yql. This causes a bad client request response (400):

In [17]:

with app.syncio(connections=12) as session:
    try:
        response: VespaQueryResponse = session.query(
            hits=1, body={"yql": "select * from sources * where userQuery()"}
        )
        print(response.is_successful())
    except VespaError as e:
        print(str(e))

with app.syncio(connections=12) as session: try: response: VespaQueryResponse = session.query( hits=1, body={"yql": "select * from sources * where userQuery()"} ) print(response.is_successful()) except VespaError as e: print(str(e))

[{'code': 3, 'summary': 'Illegal query', 'source': 'content', 'message': 'No query'}]

Using the Querybuilder DSL API¶

From pyvespa>=0.52.0, we provide a Domain Specific Language (DSL) that allows you to build queries programmatically in the vespa.querybuilder-module. See reference for full details. There are also many examples in our tests:

• https://github.com/vespa-engine/pyvespa/blob/master/tests/unit/test_grouping.py
• https://github.com/vespa-engine/pyvespa/blob/master/tests/unit/test_qb.py
• https://github.com/vespa-engine/pyvespa/blob/master/tests/integration/test_integration_grouping.py
• https://github.com/vespa-engine/pyvespa/blob/master/tests/integration/test_integration_queries.py

This section demonstrates common query patterns using the querybuilder DSL. All features of the Vespa Query Language are supported by the querybuilder DSL.

Using the Querybuilder DSL is completely optional, and you can always use the Vespa Query Language directly by passing the query as a string, which might be more convenient for simple queries.

We will use our own documentation search app for the following examples. For details of the app configuration, see the corresponding github repository.

In [18]:

app = Vespa(url="https://api.search.vespa.ai")

app = Vespa(url="https://api.search.vespa.ai")

Example 1 - matches, order by and limit¶

We want to find the 10 documents with the most terms in the 'pyvespa'-namespace (the documentation search has a 'namespace'-field, which refers to the source of the documentation). Note that the documentation search operates on the 'paragraph'-schema, but for demo purposes, we will use the 'document'-schema.

In [19]:

import vespa.querybuilder as qb
from vespa.querybuilder import QueryField

namespace = QueryField("namespace")
q = (
    qb.select(["title", "path", "term_count"])
    .from_("doc")
    .where(
        namespace.matches("pyvespa")
    )  # matches is regex-match, see https://docs.vespa.ai/en/reference/query-language-reference.html#matches
    .order_by("term_count", ascending=False)
    .set_limit(10)
)
print(f"Query: {q}")
resp = app.query(yql=q)
results = [hit["fields"] for hit in resp.hits]
df = pd.DataFrame(results)
df

import vespa.querybuilder as qb from vespa.querybuilder import QueryField namespace = QueryField("namespace") q = ( qb.select(["title", "path", "term_count"]) .from_("doc") .where( namespace.matches("pyvespa") ) # matches is regex-match, see https://docs.vespa.ai/en/reference/query-language-reference.html#matches .order_by("term_count", ascending=False) .set_limit(10) ) print(f"Query: {q}") resp = app.query(yql=q) results = [hit["fields"] for hit in resp.hits] df = pd.DataFrame(results) df

Query: select title, path, term_count from doc where namespace matches "pyvespa" order by term_count desc limit 10

Out[19]:

	path	title	term_count
0	/examples/feed_performance.html	Feeding performance	74798
1	/reference-api.html	Reference API	18282
2	/examples/simplified-retrieval-with-colpali-vl...	Scaling ColPALI (VLM) Retrieval	13943
3	/examples/pdf-retrieval-with-ColQwen2-vlm_Vesp...	PDF-Retrieval using ColQWen2 (ColPali) with Vespa	12939
4	/examples/colpali-document-retrieval-vision-la...	Vespa 🤝 ColPali: Efficient Document Retrieval ...	12666
5	/examples/colpali-benchmark-vqa-vlm_Vespa-clou...	ColPali Ranking Experiments on DocVQA	11707
6	/examples/multi-vector-indexing.html	Multi-vector indexing with HNSW	7907
7	/examples/billion-scale-vector-search-with-coh...	Billion-scale vector search with Cohere binary...	6531
8	/examples/visual_pdf_rag_with_vespa_colpali_cl...	Visual PDF RAG with Vespa - ColPali demo appli...	5666
9	/examples/chat_with_your_pdfs_using_colbert_la...	Chat with your pdfs with ColBERT, langchain, a...	5628

Example 2 - timestamp range, contains¶

We want to find the documents where one of the indexed fields contains the query term embedding,is updated after Jan 1st 2024 and the current timestamp, and have the documents ranked the 'documentation' rank profile. See https://github.com/vespa-cloud/vespa-documentation-search/blob/main/src/main/application/schemas/doc.sd.

In [20]:

import vespa.querybuilder as qb
from vespa.querybuilder import QueryField
from datetime import datetime

queryterm = "embedding"

# We need to instantiate a QueryField for fields that we want to call methods on
last_updated = QueryField("last_updated")
title = QueryField("title")
headers = QueryField("headers")
path = QueryField("path")
namespace = QueryField("namespace")
content = QueryField("content")

from_ts = int(datetime(2024, 1, 1).timestamp())
to_ts = int(datetime.now().timestamp())
print(f"From: {from_ts}, To: {to_ts}")
q = (
    qb.select(
        [title, last_updated, content]
    )  # Select takes either a list of QueryField or strings, (or '*' for all fields)
    .from_("doc")
    .where(
        namespace.matches("op.*")
        & last_updated.in_range(from_ts, to_ts)  # could also use > and <
        & qb.weakAnd(
            title.contains(queryterm),
            content.contains(queryterm),
            headers.contains(queryterm),
            path.contains(queryterm),
        )
    )
    .set_limit(3)
)
print(f"Query: {q}")
resp = app.query(yql=q, ranking="documentation")

import vespa.querybuilder as qb from vespa.querybuilder import QueryField from datetime import datetime queryterm = "embedding" # We need to instantiate a QueryField for fields that we want to call methods on last_updated = QueryField("last_updated") title = QueryField("title") headers = QueryField("headers") path = QueryField("path") namespace = QueryField("namespace") content = QueryField("content") from_ts = int(datetime(2024, 1, 1).timestamp()) to_ts = int(datetime.now().timestamp()) print(f"From: {from_ts}, To: {to_ts}") q = ( qb.select( [title, last_updated, content] ) # Select takes either a list of QueryField or strings, (or '*' for all fields) .from_("doc") .where( namespace.matches("op.*") & last_updated.in_range(from_ts, to_ts) # could also use > and < & qb.weakAnd( title.contains(queryterm), content.contains(queryterm), headers.contains(queryterm), path.contains(queryterm), ) ) .set_limit(3) ) print(f"Query: {q}") resp = app.query(yql=q, ranking="documentation")

From: 1704063600, To: 1736775672
Query: select title, last_updated, content from doc where namespace matches "op.*" and range(last_updated, 1704063600, 1736775672) and weakAnd(title contains "embedding", content contains "embedding", headers contains "embedding", path contains "embedding") limit 3

In [21]:

df = pd.DataFrame([hit["fields"] | hit for hit in resp.hits])
df = pd.concat(
    [
        df.drop(["matchfeatures", "fields"], axis=1),
        pd.json_normalize(df["matchfeatures"]),
    ],
    axis=1,
)
df.T

df = pd.DataFrame([hit["fields"] | hit for hit in resp.hits]) df = pd.concat( [ df.drop(["matchfeatures", "fields"], axis=1), pd.json_normalize(df["matchfeatures"]), ], axis=1, ) df.T

Out[21]:

	0	1	2
content	<sep />similar data by finding nearby points i...	Reference configuration for <hi>embedders</hi>...	<sep /> basic news search application - applic...
title	Embedding	Embedding Reference	News search and recommendation tutorial - embe...
last_updated	1736505422	1736505422	1736505423
id	index:documentation/0/5d6e77ca20d4e8ee29716747	index:documentation/1/a03c4aef22fcde916804d3d9	index:documentation/1/ad44f35cbd7b8214f88963e3
relevance	23.547446	22.404666	16.870303
source	documentation	documentation	documentation
bm25(content)	2.633568	2.597595	2.63281
bm25(headers)	7.572596	8.207328	5.537104
bm25(keywords)	0.0	0.0	0.0
bm25(path)	3.934699	3.293068	3.044614
bm25(title)	4.703291	4.153337	2.827888
fieldLength(content)	3830.0	2031.0	3273.0
fieldLength(title)	1.0	2.0	6.0
fieldMatch(content)	0.915766	0.892113	0.915871
fieldMatch(content).matches	1.0	1.0	1.0
fieldMatch(title)	1.0	0.933869	0.842758
query(contentWeight)	1.0	1.0	1.0
query(headersWeight)	1.0	1.0	1.0
query(pathWeight)	1.0	1.0	1.0
query(titleWeight)	2.0	2.0	2.0

Example 3 - Basic grouping¶

Vespa supports grouping and aggregation of matches through the Vespa grouping language. For an introduction to grouping, see https://docs.vespa.ai/en/grouping.html.

We will use purchase schema that is also deployed in the documentation search app.

In [22]:

from vespa.querybuilder import Grouping as G

grouping = G.all(
    G.group("customer"),
    G.each(G.output(G.sum("price"))),
)
q = qb.select("*").from_("purchase").where(True).set_limit(0).groupby(grouping)
print(f"Query: {q}")
resp = app.query(yql=q)
group = resp.hits[0]["children"][0]["children"]
# get value and sum(price) into a DataFrame
df = pd.DataFrame([hit["fields"] | hit for hit in group])
df = df.loc[:, ["value", "sum(price)"]]
df

from vespa.querybuilder import Grouping as G grouping = G.all( G.group("customer"), G.each(G.output(G.sum("price"))), ) q = qb.select("*").from_("purchase").where(True).set_limit(0).groupby(grouping) print(f"Query: {q}") resp = app.query(yql=q) group = resp.hits[0]["children"][0]["children"] # get value and sum(price) into a DataFrame df = pd.DataFrame([hit["fields"] | hit for hit in group]) df = df.loc[:, ["value", "sum(price)"]] df

Query: select * from purchase where true limit 0 | all(group(customer) each(output(sum(price))))

Out[22]:

	value	sum(price)
0	Brown	20537
1	Jones	39816
2	Smith	19484

Example 4 - Nested grouping¶

Let's find out how much each customer has spent per day by grouping on customer, then date:

In [23]:

from vespa.querybuilder import Grouping as G

# First, we construct the grouping expression:
grouping = G.all(
    G.group("customer"),
    G.each(
        G.group(G.time_date("date")),
        G.each(
            G.output(G.sum("price")),
        ),
    ),
)
# Then, we construct the query:
q = qb.select("*").from_("purchase").where(True).groupby(grouping)
print(f"Query: {q}")
resp = app.query(yql=q)
group_data = resp.hits[0]["children"][0]["children"]
records = [
    {
        "GroupId": group["value"],
        "Date": date_entry["value"],
        "Sum(price)": date_entry["fields"].get("sum(price)", 0),
    }
    for group in group_data
    for date_group in group.get("children", [])
    for date_entry in date_group.get("children", [])
]

# Create DataFrame
df = pd.DataFrame(records)
df

from vespa.querybuilder import Grouping as G # First, we construct the grouping expression: grouping = G.all( G.group("customer"), G.each( G.group(G.time_date("date")), G.each( G.output(G.sum("price")), ), ), ) # Then, we construct the query: q = qb.select("*").from_("purchase").where(True).groupby(grouping) print(f"Query: {q}") resp = app.query(yql=q) group_data = resp.hits[0]["children"][0]["children"] records = [ { "GroupId": group["value"], "Date": date_entry["value"], "Sum(price)": date_entry["fields"].get("sum(price)", 0), } for group in group_data for date_group in group.get("children", []) for date_entry in date_group.get("children", []) ] # Create DataFrame df = pd.DataFrame(records) df

Query: select * from purchase where true | all(group(customer) each(group(time.date(date)) each(output(sum(price)))))

Out[23]:

	GroupId	Date	Sum(price)
0	Brown	2006-9-10	7540
1	Brown	2006-9-11	1597
2	Brown	2006-9-8	8000
3	Brown	2006-9-9	3400
4	Jones	2006-9-10	8900
5	Jones	2006-9-11	20816
6	Jones	2006-9-8	8000
7	Jones	2006-9-9	2100
8	Smith	2006-9-10	6100
9	Smith	2006-9-11	2584
10	Smith	2006-9-6	1000
11	Smith	2006-9-7	3000
12	Smith	2006-9-9	6800

Example 5 - Grouping with expressions¶

Instead of just grouping on some attribute value, the group clause may contain arbitrarily complex expressions - see Grouping reference for exhaustive list.

Examples:

• Select the minimum or maximum of sub-expressions
• Addition, subtraction, multiplication, division, and even modulo of - sub-expressions
• Bitwise operations on sub-expressions
• Concatenation of the results of sub-expressions

Let's use some of these expressions to get the sum the prices of purchases on a per-hour-of-day basis.

In [24]:

from vespa.querybuilder import Grouping as G

grouping = G.all(
    G.group(G.mod(G.div("date", G.mul(60, 60)), 24)),
    G.order(-G.sum("price")),
    G.each(G.output(G.sum("price"))),
)
q = qb.select("*").from_("purchase").where(True).groupby(grouping)
print(f"Query: {q}")
resp = app.query(yql=q)
group_data = resp.hits[0]["children"][0]["children"]
df = pd.DataFrame([hit["fields"] | hit for hit in group_data])
df = df.loc[:, ["value", "sum(price)"]]
df

from vespa.querybuilder import Grouping as G grouping = G.all( G.group(G.mod(G.div("date", G.mul(60, 60)), 24)), G.order(-G.sum("price")), G.each(G.output(G.sum("price"))), ) q = qb.select("*").from_("purchase").where(True).groupby(grouping) print(f"Query: {q}") resp = app.query(yql=q) group_data = resp.hits[0]["children"][0]["children"] df = pd.DataFrame([hit["fields"] | hit for hit in group_data]) df = df.loc[:, ["value", "sum(price)"]] df

Query: select * from purchase where true | all(group(mod(div(date, mul(60, 60)),24)) order(-sum(price)) each(output(sum(price))))

Out[24]:

	value	sum(price)
0	10	26181
1	9	23524
2	8	22367
3	11	6765
4	7	1000