🤖 Large language models (LLMs)

Overview

Embedchain comes with built-in support for various popular large language models. We handle the complexity of integrating these models for you, allowing you to easily customize your language model interactions through a user-friendly interface.

OpenAI

Google AI

Azure OpenAI

Anthropic

Cohere

Together

Ollama

vLLM

GPT4All

JinaChat

Hugging Face

Llama2

Vertex AI

Mistral AI

AWS Bedrock

OpenAI

To use OpenAI LLM models, you have to set the OPENAI_API_KEY environment variable. You can obtain the OpenAI API key from the OpenAI Platform.

Once you have obtained the key, you can use it like this:

import os
from embedchain import App

os.environ['OPENAI_API_KEY'] = 'xxx'

app = App()
app.add("https://en.wikipedia.org/wiki/OpenAI")
app.query("What is OpenAI?")

If you are looking to configure the different parameters of the LLM, you can do so by loading the app using a yaml config file.

import os
from embedchain import App

os.environ['OPENAI_API_KEY'] = 'xxx'

# load llm configuration from config.yaml file
app = App.from_config(config_path="config.yaml")

Function Calling

To enable function calling in your application using embedchain and OpenAI, you need to pass functions into OpenAILlm class as an array of functions. Here are several ways in which you can achieve that:

Examples:

Using Pydantic Models

import os
from embedchain import App
from embedchain.llm.openai import OpenAILlm
import requests
from pydantic import BaseModel, Field, ValidationError, field_validator

os.environ["OPENAI_API_KEY"] = "sk-xxx"

class QA(BaseModel):
  """
  A question and answer pair.
  """

  question: str = Field(
      ..., description="The question.", example="What is a mountain?"
  )
  answer: str = Field(
      ..., description="The answer.", example="A mountain is a hill."
  )
  person_who_is_asking: str = Field(
      ..., description="The person who is asking the question.", example="John"
  )

  @field_validator("question")
  def question_must_end_with_a_question_mark(cls, v):
      """
      Validate that the question ends with a question mark.
      """
      if not v.endswith("?"):
          raise ValueError("question must end with a question mark")
      return v

  @field_validator("answer")
  def answer_must_end_with_a_period(cls, v):
      """
      Validate that the answer ends with a period.
      """
      if not v.endswith("."):
          raise ValueError("answer must end with a period")
      return v

llm = OpenAILlm(config=None,functions=[QA])
app = App(llm=llm)

result = app.query("Hey I am Sid. What is a mountain? A mountain is a hill.")

print(result)

Using OpenAI JSON schema

import os
from embedchain import App
from embedchain.llm.openai import OpenAILlm
import requests
from pydantic import BaseModel, Field, ValidationError, field_validator

os.environ["OPENAI_API_KEY"] = "sk-xxx"

json_schema = {
    "name": "get_qa",
    "description": "A question and answer pair and the user who is asking the question.",
    "parameters": {
        "type": "object",
        "properties": {
            "question": {"type": "string", "description": "The question."},
            "answer": {"type": "string", "description": "The answer."},
            "person_who_is_asking": {
                "type": "string",
                "description": "The person who is asking the question.",
            }
        },
        "required": ["question", "answer", "person_who_is_asking"],
    },
}

llm = OpenAILlm(config=None,functions=[json_schema])
app = App(llm=llm)

result = app.query("Hey I am Sid. What is a mountain? A mountain is a hill.")

print(result)

Using actual python functions

import os
from embedchain import App
from embedchain.llm.openai import OpenAILlm
import requests
from pydantic import BaseModel, Field, ValidationError, field_validator

os.environ["OPENAI_API_KEY"] = "sk-xxx"

def find_info_of_pokemon(pokemon: str):
  """
  Find the information of the given pokemon.
  Args:
      pokemon: The pokemon.
  """
  req = requests.get(f"https://pokeapi.co/api/v2/pokemon/{pokemon}")
  if req.status_code == 404:
      raise ValueError("pokemon not found")
  return req.json()

llm = OpenAILlm(config=None,functions=[find_info_of_pokemon])
app = App(llm=llm)

result = app.query("Tell me more about the pokemon pikachu.")

print(result)

Google AI

To use Google AI model, you have to set the GOOGLE_API_KEY environment variable. You can obtain the Google API key from the Google Maker Suite

import os
from embedchain import App

os.environ["GOOGLE_API_KEY"] = "xxx"

app = App.from_config(config_path="config.yaml")

app.add("https://www.forbes.com/profile/elon-musk")

response = app.query("What is the net worth of Elon Musk?")
if app.llm.config.stream: # if stream is enabled, response is a generator
    for chunk in response:
        print(chunk)
else:
    print(response)

Azure OpenAI

To use Azure OpenAI model, you have to set some of the azure openai related environment variables as given in the code block below:

import os
from embedchain import App

os.environ["OPENAI_API_TYPE"] = "azure"
os.environ["OPENAI_API_BASE"] = "https://xxx.openai.azure.com/"
os.environ["OPENAI_API_KEY"] = "xxx"
os.environ["OPENAI_API_VERSION"] = "xxx"

app = App.from_config(config_path="config.yaml")

You can find the list of models and deployment name on the Azure OpenAI Platform.

Anthropic

To use anthropic’s model, please set the ANTHROPIC_API_KEY which you find on their Account Settings Page.

import os
from embedchain import App

os.environ["ANTHROPIC_API_KEY"] = "xxx"

# load llm configuration from config.yaml file
app = App.from_config(config_path="config.yaml")

Cohere

Install related dependencies using the following command:

pip install --upgrade 'embedchain[cohere]'

Set the COHERE_API_KEY as environment variable which you can find on their Account settings page.

Once you have the API key, you are all set to use it with Embedchain.

import os
from embedchain import App

os.environ["COHERE_API_KEY"] = "xxx"

# load llm configuration from config.yaml file
app = App.from_config(config_path="config.yaml")

Together

Install related dependencies using the following command:

pip install --upgrade 'embedchain[together]'

Set the TOGETHER_API_KEY as environment variable which you can find on their Account settings page.

Once you have the API key, you are all set to use it with Embedchain.

import os
from embedchain import App

os.environ["TOGETHER_API_KEY"] = "xxx"

# load llm configuration from config.yaml file
app = App.from_config(config_path="config.yaml")

Ollama

Setup Ollama using https://github.com/jmorganca/ollama

import os
from embedchain import App

# load llm configuration from config.yaml file
app = App.from_config(config_path="config.yaml")

vLLM

Setup vLLM by following instructions given in their docs.

import os
from embedchain import App

# load llm configuration from config.yaml file
app = App.from_config(config_path="config.yaml")

GPT4ALL

Install related dependencies using the following command:

pip install --upgrade 'embedchain[opensource]'

GPT4all is a free-to-use, locally running, privacy-aware chatbot. No GPU or internet required. You can use this with Embedchain using the following code:

from embedchain import App

# load llm configuration from config.yaml file
app = App.from_config(config_path="config.yaml")

JinaChat

First, set JINACHAT_API_KEY in environment variable which you can obtain from their platform.

Once you have the key, load the app using the config yaml file:

import os
from embedchain import App

os.environ["JINACHAT_API_KEY"] = "xxx"
# load llm configuration from config.yaml file
app = App.from_config(config_path="config.yaml")

Hugging Face

Install related dependencies using the following command:

pip install --upgrade 'embedchain[huggingface-hub]'

First, set HUGGINGFACE_ACCESS_TOKEN in environment variable which you can obtain from their platform.

Once you have the token, load the app using the config yaml file:

import os
from embedchain import App

os.environ["HUGGINGFACE_ACCESS_TOKEN"] = "xxx"

# load llm configuration from config.yaml file
app = App.from_config(config_path="config.yaml")

Custom Endpoints

You can also use Hugging Face Inference Endpoints to access custom endpoints. First, set the HUGGINGFACE_ACCESS_TOKEN as above.

Then, load the app using the config yaml file:

import os
from embedchain import App

os.environ["HUGGINGFACE_ACCESS_TOKEN"] = "xxx"

# load llm configuration from config.yaml file
app = App.from_config(config_path="config.yaml")

If your endpoint requires additional parameters, you can pass them in the model_kwargs field:

llm:
  provider: huggingface
  config:
    endpoint: <YOUR_ENDPOINT_URL_HERE>
    model_kwargs:
      max_new_tokens: 100
      temperature: 0.5

Currently only supports text-generation and text2text-generation for now [ref].

See langchain’s hugging face endpoint for more information.

Llama2

Llama2 is integrated through Replicate. Set REPLICATE_API_TOKEN in environment variable which you can obtain from their platform.

Once you have the token, load the app using the config yaml file:

import os
from embedchain import App

os.environ["REPLICATE_API_TOKEN"] = "xxx"

# load llm configuration from config.yaml file
app = App.from_config(config_path="config.yaml")

Vertex AI

Setup Google Cloud Platform application credentials by following the instruction on GCP. Once setup is done, use the following code to create an app using VertexAI as provider:

from embedchain import App

# load llm configuration from config.yaml file
app = App.from_config(config_path="config.yaml")

Mistral AI

Obtain the Mistral AI api key from their console.

os.environ["MISTRAL_API_KEY"] = "xxx"

app = App.from_config(config_path="config.yaml")

app.add("https://www.forbes.com/profile/elon-musk")

response = app.query("what is the net worth of Elon Musk?")
# As of January 16, 2024, Elon Musk's net worth is $225.4 billion.

response = app.chat("which companies does elon own?")
# Elon Musk owns Tesla, SpaceX, Boring Company, Twitter, and X.

response = app.chat("what question did I ask you already?")
# You have asked me several times already which companies Elon Musk owns, specifically Tesla, SpaceX, Boring Company, Twitter, and X.

AWS Bedrock

Setup

Before using the AWS Bedrock LLM, make sure you have the appropriate model access from Bedrock Console.
You will also need to authenticate the boto3 client by using a method in the AWS documentation
You can optionally export an AWS_REGION

Usage

import os
from embedchain import App

os.environ["AWS_ACCESS_KEY_ID"] = "xxx"
os.environ["AWS_SECRET_ACCESS_KEY"] = "xxx"
os.environ["AWS_REGION"] = "us-west-2"

app = App.from_config(config_path="config.yaml")

The model arguments are different for each providers. Please refer to the AWS Bedrock Documentation to find the appropriate arguments for your model.

If you can't find the specific LLM you need, no need to fret. We're continuously expanding our support for additional LLMs, and you can help us prioritize by opening an issue on our GitHub or simply reaching out to us on our Slack or Discord community.

Slack

Let us know on our slack community

Discord

Let us know on discord community

GitHub

Open an issue on our GitHub

Schedule a call

Schedule a call with Embedchain founder

Get Started

Use cases

Components

Deployment

Community

Contributing

Product

​Overview

OpenAI

Google AI

Azure OpenAI

Anthropic

Cohere

Together

Ollama

vLLM

GPT4All

JinaChat

Hugging Face

Llama2

Vertex AI

Mistral AI

AWS Bedrock

​OpenAI

​Function Calling

​Google AI

​Azure OpenAI

​Anthropic

​Cohere

​Together

​Ollama

​vLLM

​GPT4ALL

​JinaChat

​Hugging Face

​Custom Endpoints

​Llama2

​Vertex AI

​Mistral AI

​AWS Bedrock

​Setup

​Usage

Slack

Discord

GitHub

Schedule a call

Overview

OpenAI

Function Calling

Google AI

Azure OpenAI

Anthropic

Cohere

Together

Ollama

vLLM

GPT4ALL

JinaChat

Hugging Face

Custom Endpoints

Llama2

Vertex AI

Mistral AI

AWS Bedrock

Setup

Usage