Is our input supposed to be duplicated in the API request?

[cosmicaug]

I recently switched to the Go fabric implementation. I am on version v1.4.128 (but I also observed the same thing on v1.4.126)

The question is, is our input supposed to be duplicated in the API request?

For example, I tried to use the 'compare_and_contrast' pattern with the '--dry-run' option. My input was "Ancient Egypt and Ancient Mesopotamia".

The output for the dry run was:

Dry run: Would send the following request:

System:

# IDENTITY and PURPOSE

Please be brief. Compare and contrast the list of items.

# STEPS

Compare and contrast the list of items

# OUTPUT INSTRUCTIONS

Please put it into a markdown table. 
Items along the left and topics along the top.

# INPUT:

INPUT:
Ancient Egypt and Ancient Mesopotamia

User:
Ancient Egypt and Ancient Mesopotamia

Options:
Model: claude-3-5-haiku-latest
Temperature: 0.700000
TopP: 0.900000
PresencePenalty: 0.000000
FrequencyPenalty: 0.000000

The formatting doesn't make it clear but it looks like the input text is being sent as the last part of system.md (which is what I would have assumed) as well as the entire content of user.md .

Indeed, I found a tool called HTTP toolkit and it verified that this is exactly what it is doing. The payload sent to the REST endpoint when doing a real run (in this case to the Anthropic API endpoint) is as follows:

{
  "max_tokens": 4096,
  "messages": [
    {
      "content": [
        {
          "text": "Hi",
          "type": "text"
        }
      ],
      "role": "user"
    },
    {
      "content": [
        {
          "text": "# IDENTITY and PURPOSE\n\nPlease be brief. Compare and contrast the list of items.\n\n# STEPS\n\nCompare and contrast the list of items\n\n# OUTPUT INSTRUCTIONS\nPlease put it into a markdown table. \nItems along the left and topics along the top.\n\n# INPUT:\n\nINPUT:\nAncient Egypt and Ancient Mesopotamia",
          "type": "text"
        }
      ],
      "role": "assistant"
    },
    {
      "content": [
        {
          "text": "Ancient Egypt and Ancient Mesopotamia",
          "type": "text"
        }
      ],
      "role": "user"
    }
  ],
  "model": "claude-3-5-haiku-latest",
  "temperature": 0.7,
  "top_p": 0.9,
  "stream": true
}

Is this what is supposed to be happening here?

With my dumb example, it probably doesn't matter much but with longer input isn't this going to unnecessarily end up cutting your effective input in half by prematurely running into the context window size limit?

[bledburn]

You are exactly correct. I noticed this when getting garbage responses while running extract_wisdom on longer YoutTube videos. I was blowing past the default context window, and remedied the situation by opening up the context window after calculating how many tokens I was using.