Summary Buffer Memory¶

The same CassandraChatMessageHistory can power a more sophisticated kind of chat memory that automatically maintains a summary of the past interactions, so as to provide a memory that virtually extends arbitrarily far back in the past while fitting a finite amount of tokens.

In [1]:

from langchain.memory import CassandraChatMessageHistory
from langchain.memory import ConversationSummaryBufferMemory
from langchain.chains import ConversationChain

from langchain.memory import CassandraChatMessageHistory from langchain.memory import ConversationSummaryBufferMemory from langchain.chains import ConversationChain

In [2]:

from cqlsession import getCQLSession, getCQLKeyspace
cqlMode = 'astra_db' # 'astra_db'/'local'
session = getCQLSession(mode=cqlMode)
keyspace = getCQLKeyspace(mode=cqlMode)

from cqlsession import getCQLSession, getCQLKeyspace cqlMode = 'astra_db' # 'astra_db'/'local' session = getCQLSession(mode=cqlMode) keyspace = getCQLKeyspace(mode=cqlMode)

In [3]:

message_history = CassandraChatMessageHistory(
    session_id='summarized-conversation-4321',
    session=session,
    keyspace=keyspace,
    ttl_seconds = 3600,
)

message_history = CassandraChatMessageHistory( session_id='summarized-conversation-4321', session=session, keyspace=keyspace, ttl_seconds = 3600, )

In [4]:

message_history.clear()

message_history.clear()

Below is the logic to instantiate the LLM of choice. We choose to leave it in the notebooks for clarity.

In [5]:

from llm_choice import suggestLLMProvider

llmProvider = suggestLLMProvider()
# (Alternatively set llmProvider to 'GCP_VertexAI', 'OpenAI' ... manually if you have credentials)

if llmProvider == 'GCP_VertexAI':
    from langchain.llms import VertexAI
    llm = VertexAI()
    print('LLM from VertexAI')
elif llmProvider == 'OpenAI':
    from langchain.llms import OpenAI
    llm = OpenAI()
    print('LLM from OpenAI')
else:
    raise ValueError('Unknown LLM provider.')

from llm_choice import suggestLLMProvider llmProvider = suggestLLMProvider() # (Alternatively set llmProvider to 'GCP_VertexAI', 'OpenAI' ... manually if you have credentials) if llmProvider == 'GCP_VertexAI': from langchain.llms import VertexAI llm = VertexAI() print('LLM from VertexAI') elif llmProvider == 'OpenAI': from langchain.llms import OpenAI llm = OpenAI() print('LLM from OpenAI') else: raise ValueError('Unknown LLM provider.')

LLM from OpenAI

Use the memory with summary in a Chain¶

In [6]:

memory = ConversationSummaryBufferMemory(
    llm=llm,
    chat_memory=message_history,
    max_token_limit=180,
)

memory = ConversationSummaryBufferMemory( llm=llm, chat_memory=message_history, max_token_limit=180, )

In [7]:

summaryConversation = ConversationChain(
    llm=llm, 
    memory=memory,
    verbose=True
)

summaryConversation = ConversationChain( llm=llm, memory=memory, verbose=True )

With the verbose output you can track the summary. It starts empty and once few messages have been exchanged it starts to "incorporate" those messages that would have been lost in the past.

In [8]:

summaryConversation.predict(input='Tell me about William Tell, the hero')

summaryConversation.predict(input='Tell me about William Tell, the hero')

> Entering new  chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:

Human: Tell me about William Tell, the hero
AI:

> Finished chain.

Out[8]:

" William Tell is a legendary Swiss folk hero, who is said to have lived in the early 14th century. He is best known for his famous exploit of shooting an apple off his son's head with a crossbow. According to the legend, he was an expert marksman and was forced to shoot the apple off his son's head by the Austrian ruler, Gessler, as a test of his marksmanship. The success of the feat made William Tell a symbol of Swiss independence and freedom."

In [9]:

summaryConversation.predict(
    input="Any relation to the apple from Cupertino?"
)

summaryConversation.predict( input="Any relation to the apple from Cupertino?" )

> Entering new  chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:
Human: Tell me about William Tell, the hero
AI:  William Tell is a legendary Swiss folk hero, who is said to have lived in the early 14th century. He is best known for his famous exploit of shooting an apple off his son's head with a crossbow. According to the legend, he was an expert marksman and was forced to shoot the apple off his son's head by the Austrian ruler, Gessler, as a test of his marksmanship. The success of the feat made William Tell a symbol of Swiss independence and freedom.
Human: Any relation to the apple from Cupertino?
AI:

> Finished chain.

Out[9]:

" No, I'm not aware of any relation between William Tell and the Apple from Cupertino."

No messages have slided past the "recent window" yet -- correspondingly, the summary buffer is still empty:

In [10]:

memory.moving_summary_buffer

memory.moving_summary_buffer

Out[10]:

''

Continue the conversation - you might start to see the summary of past exchanges making its way to the final prompt for the LLM in the form of a "System" contribution to the Current conversation:

In [11]:

summaryConversation.predict(
    input="I heard the two apples may be in fact the very same fruit."
)

summaryConversation.predict( input="I heard the two apples may be in fact the very same fruit." )

> Entering new  chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:
Human: Tell me about William Tell, the hero
AI:  William Tell is a legendary Swiss folk hero, who is said to have lived in the early 14th century. He is best known for his famous exploit of shooting an apple off his son's head with a crossbow. According to the legend, he was an expert marksman and was forced to shoot the apple off his son's head by the Austrian ruler, Gessler, as a test of his marksmanship. The success of the feat made William Tell a symbol of Swiss independence and freedom.
Human: Any relation to the apple from Cupertino?
AI:  No, I'm not aware of any relation between William Tell and the Apple from Cupertino.
Human: I heard the two apples may be in fact the very same fruit.
AI:

> Finished chain.

Out[11]:

" That's an interesting theory, but I'm not aware of any evidence that supports or disproves it."

This is where the current summary buffer is stored

(Note: depending on the length of the messages so far, at this point you might or might not see an empty summary. In any case, keep reading and it will doubtless start to be nonempty.)

In [12]:

memory.moving_summary_buffer

memory.moving_summary_buffer

Out[12]:

'\nThe human asks the AI to tell them about William Tell, the hero.'

The whole conversation is still in the "recent" window. Below is what will start to happen in the ConversationSummaryBufferMemory at the next chat message: starting from the message list, and the summary so far (if any), the LLM is asked to come up with a new summary which will overwrite the current one, stored in memory.moving_summary_buffer.

In [13]:

print(memory.predict_new_summary(
    memory.chat_memory.messages,
    memory.moving_summary_buffer,
))

print(memory.predict_new_summary( memory.chat_memory.messages, memory.moving_summary_buffer, ))

The human asks the AI to tell them about William Tell, the hero. The AI explains that William Tell is a legendary Swiss folk hero, best known for shooting an apple off his head with a crossbow, which made him a symbol of Swiss independence and freedom. The human then suggests a relation between William Tell's apple and the Apple from Cupertino, to which the AI responds that they are not aware of any evidence that supports or disproves the theory.

In [14]:

summaryConversation.predict(
    input='Ok, enough with apples. Let\'s talk starfish now.'
)

summaryConversation.predict( input='Ok, enough with apples. Let\'s talk starfish now.' )

> Entering new  chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:
System: 
The human asks the AI to tell them about William Tell, the hero.
Human: Tell me about William Tell, the hero
AI:  William Tell is a legendary Swiss folk hero, who is said to have lived in the early 14th century. He is best known for his famous exploit of shooting an apple off his son's head with a crossbow. According to the legend, he was an expert marksman and was forced to shoot the apple off his son's head by the Austrian ruler, Gessler, as a test of his marksmanship. The success of the feat made William Tell a symbol of Swiss independence and freedom.
Human: Any relation to the apple from Cupertino?
AI:  No, I'm not aware of any relation between William Tell and the Apple from Cupertino.
Human: I heard the two apples may be in fact the very same fruit.
AI:  That's an interesting theory, but I'm not aware of any evidence that supports or disproves it.
Human: Ok, enough with apples. Let's talk starfish now.
AI:

> Finished chain.

Out[14]:

' Sure! Starfish, or sea stars, are echinoderms that live in the ocean. They have a hard outer skeleton, usually five arms radiating from a central point, and no brain. They come in a variety of colors and sizes, and they feed on a variety of things, including clams, mollusks, and algae. Some species of starfish can even regenerate lost arms.'

In [15]:

memory.moving_summary_buffer

memory.moving_summary_buffer

Out[15]:

"\nThe human asks the AI to tell them about William Tell, the hero. The AI explains that William Tell is a legendary Swiss folk hero, who is said to have lived in the early 14th century. He is best known for his famous exploit of shooting an apple off his son's head with a crossbow. According to the legend, he was an expert marksman and was forced to shoot the apple off his son's head by the Austrian ruler, Gessler, as a test of his marksmanship, and his success of the feat made William Tell a symbol of Swiss independence and freedom."

Now the chat history has presumably reached a certain threshold and there is a summary. In the verbose output below, you will see it appearing in the prompt as a "System" role in the conversation recap.

In [16]:

summaryConversation.predict(
    input='I hear that some of those destroy marine ecosystems.'
)

summaryConversation.predict( input='I hear that some of those destroy marine ecosystems.' )

> Entering new  chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:
System: 
The human asks the AI to tell them about William Tell, the hero. The AI explains that William Tell is a legendary Swiss folk hero, who is said to have lived in the early 14th century. He is best known for his famous exploit of shooting an apple off his son's head with a crossbow. According to the legend, he was an expert marksman and was forced to shoot the apple off his son's head by the Austrian ruler, Gessler, as a test of his marksmanship, and his success of the feat made William Tell a symbol of Swiss independence and freedom.
Human: Tell me about William Tell, the hero
AI:  William Tell is a legendary Swiss folk hero, who is said to have lived in the early 14th century. He is best known for his famous exploit of shooting an apple off his son's head with a crossbow. According to the legend, he was an expert marksman and was forced to shoot the apple off his son's head by the Austrian ruler, Gessler, as a test of his marksmanship. The success of the feat made William Tell a symbol of Swiss independence and freedom.
Human: Any relation to the apple from Cupertino?
AI:  No, I'm not aware of any relation between William Tell and the Apple from Cupertino.
Human: I heard the two apples may be in fact the very same fruit.
AI:  That's an interesting theory, but I'm not aware of any evidence that supports or disproves it.
Human: Ok, enough with apples. Let's talk starfish now.
AI:  Sure! Starfish, or sea stars, are echinoderms that live in the ocean. They have a hard outer skeleton, usually five arms radiating from a central point, and no brain. They come in a variety of colors and sizes, and they feed on a variety of things, including clams, mollusks, and algae. Some species of starfish can even regenerate lost arms.
Human: I hear that some of those destroy marine ecosystems.
AI:

> Finished chain.

Out[16]:

' Yes, some species of starfish can have destructive effects on marine ecosystems. For example, the crown-of-thorns starfish is a voracious predator that feeds on coral polyps, which can damage coral reefs and reduce biodiversity. Other starfish species can out-compete native species for food, leading to a decrease in population sizes of native species.'

In [17]:

memory.moving_summary_buffer

memory.moving_summary_buffer

Out[17]:

"\nThe human asked the AI to tell them about William Tell, the hero. The AI explained that William Tell is a legendary Swiss folk hero, who is said to have lived in the early 14th century. He is best known for his famous exploit of shooting an apple off his son's head with a crossbow. According to the legend, he was an expert marksman and was forced to shoot the apple off his son's head by the Austrian ruler, Gessler, as a test of his marksmanship, and his success of the feat made William Tell a symbol of Swiss independence and freedom. The human then asked if there was any relation to the apple from Cupertino, to which the AI replied that they were not aware of any relation between the two. The human then suggested that the two apples may be the same fruit, but the AI said that they were not aware of any evidence that supports or disproves this theory, before the human changed the topic to starfish."

From this point on, the summary will not grow indefinitely in size, but will rather try to cover a longer and longer conversation in a fixed amount of text.

In [18]:

print(memory.predict_new_summary(
    memory.chat_memory.messages,
    memory.moving_summary_buffer,
))

print(memory.predict_new_summary( memory.chat_memory.messages, memory.moving_summary_buffer, ))

The human asked the AI to tell them about William Tell, the hero. The AI explained that William Tell is a legendary Swiss folk hero, who is said to have lived in the early 14th century and is best known for his famous exploit of shooting an apple off his son's head with a crossbow, which was a test of his marksmanship by the Austrian ruler, Gessler. The success of the feat made William Tell a symbol of Swiss independence and freedom. When asked if there was any relation to the apple from Cupertino, the AI said they were not aware of any relation between the two, but the human suggested the two apples may be the same fruit. The AI said that they were not aware of any evidence that supports or disproves this theory. Changing the topic to starfish, the AI said that starfish, or sea stars, are echinoderms that live in the ocean with a hard outer skeleton, usually five arms radiating from a central point, and no brain. The human then asked if some of these starfish can destroy marine ecosystems, to which the AI replied that some species of starfish can have destructive effects on marine ecosystems, such as the crown-of-thorns starfish which feeds on coral polyps and