Cannot simultaneously use tools (bindTools to bind) and structured output (withStructuredOutput) with Langchainjs

[ameramayreh]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

import axios from 'axios';
import { OpenAI } from "openai";
import { ChatCompletionMessageParam, ChatCompletionTool } from "openai/resources";
import { zodResponseFormat } from "openai/helpers/zod";
import { z } from "zod";

/**
 * The response format for the weather query response.
 */
const responseFormat = z.object({
  responseMessage: z.string().describe("The response message"),
  isUserQueryAnswered: z.boolean().describe("Whether the user query is answered"),
  lastWeatherLocation: z.string().optional().describe("The last location for which weather data was fetched"),
  lastTemperatureUnit: z.enum(["C", "F"]).optional().describe("The last temperature unit"),
});


/**
 * [Tool Service] Fetches weather forecast data from Open-Meteo API.
 * @param latitude - Latitude of the location.
 * @param longitude - Longitude of the location.
 * @returns A promise that resolves to the weather data.
 */
async function fetchWeatherData(latitude: number, longitude: number): Promise<string> {
  try {
    const response = await axios.get('https://api.open-meteo.com/v1/forecast', {
      params: {
        latitude,
        longitude,
        current: 'temperature_2m,relative_humidity_2m,precipitation,rain',
        hourly: 'temperature_2m,precipitation_probability,precipitation',
        daily: 'temperature_2m_max,temperature_2m_min,precipitation_sum',
        forecast_days: 1,
      },
    });

    console.log('Weather data:', response.data);

    return JSON.stringify(response.data);
  } catch (error) {
    console.error('Error fetching weather data:', error);
    throw error;
  }
}

const toolsRegistry: { [key: string]: (params: any) => Promise<any> } = {
  "get_weather": async (params: { latitude: number, longitude: number }) => {
    return await fetchWeatherData(params.latitude, params.longitude);
  }
};

const tools = [{
  "type": "function",
  "function": {
    "name": "get_weather",
    "description": "Get current temperature for a given location.",
    "parameters": {
      "type": "object",
      "properties": {
        "latitude": {
          "type": "number",
          "description": "the latitude of the location"
        },
        "longitude": {
          "type": "number",
          "description": "the longitude of the location"
        }
      },
      "required": [
        "latitude",
        "longitude"
      ],
      "additionalProperties": false
    },
    "strict": true
  }
} as ChatCompletionTool];


const messages: ChatCompletionMessageParam[] = [];
const openai = new OpenAI();

function invoke(userQuery?: string) {
  if(userQuery && !messages.length) {
    messages.push({ role: "user", content: userQuery});
  }
  openai.chat.completions.create({
    model: "gpt-4o",
    messages: messages,
    /* Tools to be used by LLM */ 
    tools: tools,
    /* The response format (simply extract some data in addition to the response message) */ 
    response_format: zodResponseFormat(responseFormat, "WeatherQueryResponse"),
  }).then(async (completion) => {
    const message = completion.choices[0].message;
    messages.push(message);
    if (!message.tool_calls?.length) {
      console.log(message);
      return;
    }

    for (const toolCall of message.tool_calls) {
      const tool = toolsRegistry[toolCall.function.name];
      const result = await tool(JSON.parse(toolCall.function.arguments));
      messages.push({
        role: "tool",
        tool_call_id: toolCall.id,
        content: result.toString()
      });
    }
    invoke();
  });
}

invoke("What is the weather like in London today?");

Description

I'm trying to use tools and structured output with ChatOpenAI simultaneously but each method to support tools or structured outputs returns a runnable that does not support the other methods.

This works with OpenAI official SDK (see the example), but not for langchainjs (see below).

const model = new ChatOpenAI({
  model: "gpt-4o-mini",
  temperature: 0,
});

// Non of those works
const llm = model.bindTools([getWeather]).withStructuredOutput(weatherResponse);
const llm = model.bind(nativeTools).withStructuredOutput(weatherResponse);
const llm= model.withStructuredOutput(weatherResponse).bindTools([getWeather]);

Is there a way to implement the example in langchainjs?

System Info

[email protected] | MIT | deps: 12 | versions: 310
Typescript bindings for langchain
https://github.com/langchain-ai/langchainjs/tree/main/langchain/

keywords: llm, ai, gpt3, chain, prompt, prompt engineering, chatgpt, machine learning, ml, openai, embeddings, vectorstores

dist
.tarball: https://registry.npmjs.org/langchain/-/langchain-0.3.15.tgz
.shasum: c6c8b17bf20c870795e31515f48abde7d13ccc9d
.integrity: sha512-+DQ4I2iy4b5sErkxo6jAkgmumvhgqLwLB2fmiGl3yDt8+VVZdB1MUULZMzf+6ubarNc7Mwn/sxHUqK4GhEndhg==
.unpackedSize: 2.9 MB

dependencies:
@langchain/openai: >=0.1.0 <0.5.0 jsonpointer: ^5.0.1 uuid: ^10.0.0
@langchain/textsplitters: >=0.0.0 <0.2.0 langsmith: >=0.2.8 <0.4.0 yaml: ^2.2.1
js-tiktoken: ^1.0.12 openapi-types: ^12.1.3 zod-to-json-schema: ^3.22.3
js-yaml: ^4.1.0 p-retry: 4 zod: ^3.22.4

published 2 weeks ago by jacoblee93

Windows 10 pro
Node: v20.12.2
Npm: 10.5.0

[dosubot[bot]]

@ameramayreh Hello there! I'm here to help you out with any bugs, questions, or contributions you might have. Let's crush those bugs together!

In LangChainJS, the bindTools and withStructuredOutput methods are designed to be used separately, and currently, they do not support being chained together directly. This is a known limitation in the framework. The bindTools method is used to bind tools to a chat model, while withStructuredOutput configures the model to produce structured output based on a schema. However, each method returns a runnable that does not support the other method, which is why chaining them as shown in your example does not work ^[1].

To achieve the functionality you are looking for, you might need to implement a custom solution that manually handles both tool invocation and structured output parsing. This could involve invoking the tools separately and then applying the structured output schema to the results manually.

For now, you can consider handling the tool calls and structured output separately in your application logic, ensuring that the results from the tools are processed according to your structured output requirements before being passed back to the model or user. This approach would require more manual handling but can achieve the desired outcome until the framework potentially supports this feature natively in future updates.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[ameramayreh]

Thanks @dosu, how to handle this in the application logic as you suggested?

[dosubot[bot]]

To handle tool invocation and structured output parsing separately in your application logic using LangChainJS, you can follow these steps:

1. Tool Invocation: Use the bindTools method to bind your tools to the language model. You can then manually invoke these tools when needed. For example, you can maintain a registry of tools and call them based on the tool calls returned by the model.

2. Structured Output Parsing: Use the withStructuredOutput method to define the schema for your structured output. After invoking the tools and obtaining their results, you can manually apply this schema to parse and validate the output.

Here's a simplified approach:

• Define Tools and Schema: Set up your tools and structured output schema separately.

• Invoke Tools: When the model returns a tool call, use your tool registry to invoke the appropriate tool and get the result.

• Parse Output: After obtaining the tool results, apply the structured output schema to parse and validate the results.

Here's a conceptual example:

// Define your tools
const toolsRegistry = {
  get_weather: async (params) => {
    // Fetch weather data
  },
};

// Define your structured output schema
const responseFormat = z.object({
  responseMessage: z.string(),
  isUserQueryAnswered: z.boolean(),
  lastWeatherLocation: z.string().optional(),
  lastTemperatureUnit: z.enum(["C", "F"]).optional(),
});

// Function to handle tool invocation and structured output
async function handleQuery(userQuery: string) {
  // Invoke the model with the user query
  const completion = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: userQuery }],
    tools: tools, // Bind tools here
  });

  const message = completion.choices[0].message;

  // Check for tool calls
  if (message.tool_calls?.length) {
    for (const toolCall of message.tool_calls) {
      const tool = toolsRegistry[toolCall.function.name];
      const result = await tool(JSON.parse(toolCall.function.arguments));

      // Process the tool result
      const parsedResult = responseFormat.parse({
        responseMessage: result.toString(),
        isUserQueryAnswered: true, // Example logic
      });

      console.log(parsedResult);
    }
  } else {
    console.log(message);
  }
}

// Example usage
handleQuery("What is the weather like in London today?");

This approach allows you to handle tool invocation and structured output parsing separately, ensuring that each part of the process is managed according to your application's needs.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}