Working with a simple CSV in AI Resources

Overview

AI Resources enable you to ingest & transform data in a myriad of ways for use in your AI Agents or AI Services. This guide will show you how to upload, transform, and index a simple dataset for use in your AI Agent.

📘

This guide requires the use of some Python for the Transformation steps, all of which can be copied and modified from the guide itself.

Getting Started

To complete this tutorial you can either use our sample Quiqsilver Bikes FAQ CSV or create s CSV that contains Question and Answer columns and some data.

First you'll need to navigate to the Resources option in AI Studio:

Uploading your Dataset

From there you'll be taken into the Datasets, where you'll use the Upload Dataset button to upload your CSV:

Give your Dataset both a Namespace (if you haven't created one already) and a Name:

the Namespace will be the section your Dataset lives in within Datasets , and the Name will be how you reference your specific dataset:

Once you've uploaded your CSV you'll see an inferred schema, if you're using the example above you'll get 40 records, as well as an /Answer & /Question property, both of which are strings:

📘

Dataset Synchronization

While this guide works off a simple CSV upload, for production use cases you'll most likely have a more complex data schema and our data sync API. AI Resources supports both methods of data ingestion, you can learn more about how that works here.

Building your Transformation Engine

Once you've got your data into AI Resources, you may want to tranfsorm or enrich your dataset. There are a bunch of different things you can do to Transform a dataset, see Dataset Transformations for additional examples.

For the purpose of this demo, we're going create a transformation that enriches our dataset with a set of questions that will help our AI agent with search accuracy.

A transformation is simply a pipeline of steps. Each step takes in data (a row in our csv) and returns data that is passed onto the next step. A step is either a call to a LLM, a simple python function or a call to another transformation you have already built in AI Studio.

Let's get started, navigate to Transformations and select New Transformation Engine:

Name your new Transformation Engine QuiqSilver knowledge engine, or something similar:

Next, select Create New Draft, which will take you into your Transformation view:

Creating a Transformation

Next, select Add New:

and create a new Transformation called enrich:

Creating Steps

Next, you'll want to select Add Step to create the first step within your transformation, click back over to the Configuration tab and select Add Bootstrapped Step.

In this step, we will generate related questions for a each Question and Answer pair in our dataset. Set the Type to Prompt AI, LLM to gpt-4o-mini, and identifier to generate_questions.

📘

Manual Steps vs Bootstrapped Steps

A Manual Step should be used when you already have functions you'd like to use in the Code tab, a Bootstrapped Step will create the signatures of the function(s) for you:

This will bootstrap two functions:

• one to build a prompt and send the request to the LLM
• another to handle the completion returned from the LLM

Once you click save, you will be automatically navigated to the code editing panel and you should see code like this.

After reviewing the bootstrapped code, go ahead and modify it with something like this:

def build_generate_questions(llm, data):
  
  instructions = f"""
Your job is to carefully review the following question-answer-pair and then generate a set of related questions. 

<question-answer-pair>
{data['Question']}
{data['Answer']}
</question-answer-pair>

## RESPONSE 
Generate the most likely short questions that may be addressed by the question-answer-pair. 
Do your best to vary the questions, verbs, and adjectives.
Create 8 short questions.

Format the JSON results like this: 
{{
  "questions": [
    "question_1",
    "question_2",
    "question...8"
  ]
}}
JSON: """

  # set request to the request format for the LLM you are using
  return {
    "request": {
      "messages": [{"role": "user", "content": instructions}],
      "temperature": 0,
      "max_tokens": 512,
      "response_format": {
        "type": "json_object"
      }
    }
  }
   
def handle_generate_questions(llm, data, prompt, response, completion):
  
  data['questions'] = quiq.json.loads(completion)['questions']
  
  return [data]

Testing your step

Let's test our step:

• First, ensure you've saved your changes before testing using the save button in the top right.
• Next, ensure you've selected the correct Dataset and Transformation for testing:

Select the Test button to run your Transformation. you should see your step and the transformed documents.

In addition to seeing the questions that were added, you can expand the Debug Info section to see the prompt sent and completion returned from the LLM:

📘

Advanced Configuration

You can use the Advanced Configuration button next to the Test button to modify how many documents you test, as well as what document you start your test on:

Depending on your use case and data, you can add additional steps to prepare your dataset for use in your AI agent. See Dataset Transformations for common use cases.

Creating a Search Key

Now that your data is enriched, we'll want to add a field (column) that will be searched by your AI agent. For the lesson we'll call this column search_key, but you could call it embedding_text or any other name you choose. You'll want to join together all the semantically relevant text into a single field that will be embedded and searched by your AI agent. Your transformation steps normally one of two objectives:

• Help your AI agent find the right data
• Help your AI agent use that data in prompts

To create our search_key, go to code section and add this function.

def create_search_key(data):

  parts = data['questions']
  parts.append(data['Question'])
  parts.append(data['Answer'])
  data['search_key'] = '\n'.join(parts)

  return [data]

Go to the configuration tab and add a new step. Choose a type of Call Function and select create_search_key function.

For each of our steps, select Add Description and update them like so:

Save and Test your transformation. you should results that look like this:

Using your Transformation

Now that we've run through our transformation step and done some testing, we can go ahead and publish our transformation using the publish button in the upper right:

Run Transformation

Next, navigate back to your initial dataset and select the Run Transformation option from the Actions Menu:

Now, select your Transformation. You can use the same Namespace you created earlier, and give your soon to be created dataset a new name that distinguishes it from what you started with, like quiqsilver-bikes-faqs-enriched:

Once you're transformation is completed, you can view your new Dataset, take note that it now includes the properties added during our transformation step:

Creating an Index

Now that we've got our newly transformed dataset available, we'll need to create an embedding and an index for it. Go ahead and select the Add Index Button.

Next select your Embedding Model (the model used to search, the default selection works great for our purposes), and the Embedding Property (the property an AI Agent will search against) , in our case we'll use our the search_key which we just created for this purpose:

📘

Filter Properties

Depending on your dataset it may be useful to filter a given index based on a specific property, maybe you have knowledge base articles for customers and bicycle shop owners, and wanted to create separate indexes for each, you could do that by utilizing a property that was part of your initial dataset, or was added during the transformation process.

Once that's done, you'll see your newly created Index:

📘

Multiple Indexes

You can create multiple indexes for an individual dataset, for example if you're working with a product catalog you may want to have different datasets for different price points, or for different categories of products.

Creating a Search Resource

In order for an Index to be usable by an AI Agent, it needs to be associated with a Search Resource. Navigate over to the Search Resources section and create a new Search Resource:

Next, give your Search Resource a name, and select the Dataset and Index you've created:

Nice work! You've now created your first Search Resource with your newly transformed Dataset. You can ask questions of your Search Resource to see what articles get returned, and you can click into each result to see more details:

Summary

👍

Datasets

Datasets can be uploaded to AI Resources manually, or they can be synced via API. Datasets are the core building blocks that your AI Agents will use to generate answers, recommend products, and more.

👍

Transformations

Transformations enable you to modify your datasets to make them maximally usable for your AI Agent, all while keeping them in sync with the original source. Some transformation examples include:

• Removing unnecessary information from a dataset, like a phone number or social media handle from a knowledge base article
• Stripping out unwanted JSON
• Creating a topic for each article for better classification
• Creating related questions for better search results

You can use multiple steps within each transformation, and transformation engines can contain multiple transformations.

👍

Indexes

Indexes enable you to search your dataset via a property, and filter down based on properties as well.

👍

Search Resources

Search Resources enable you to use your Indexes in your AI Agents.

What's Next

Next, you'll need to integrate your newly created Search Resource into your AI Agent, you can refer to the Building a Simple Retrieval Augmented Generation Flow guide for help getting started with.

References

Code Samples

Below you'll find all of the code used in this transformation for easy reference:

def build_generate_questions(llm, data):
  
  instructions = f"""
Your job is to carefully review the following question-answer-pair and then generate a set of related questions. 

<question-answer-pair>
{data['Question']}
{data['Answer']}
</question-answer-pair>

## RESPONSE 
Generate the most likely short questions that may be addressed by the question-answer-pair. 
Do your best to vary the questions, verbs, and adjectives.
Create 8 short questions.

Format the JSON results like this: 
{{
  "questions": [
    "question_1",
    "question_2",
    "question...6"
  ]
}}
JSON: """

  # set request to the request format for the LLM you are using
  return {
    "request": {
      "messages": [{"role": "user", "content": instructions}],
      "temperature": 0,
      "max_tokens": 512,
      "response_format": {
        "type": "json_object"
      }
    }
  }
  
  
def handle_generate_questions(llm, data, prompt, response, completion):
  
  data['questions'] = quiq.json.loads(completion)['questions']
  
  return [data]
  
def create_search_key(data):

  parts = data['questions']
  parts.append(data['Question'])
  parts.append(data['Answer'])
  data['search_key'] = '\n'.join(parts)

  return [data]
  

Sample Artilces

If for any reason you're unable to access the sample CSV provided above, you can copy the content in table format here: