Enhancing AI Apps with Streaming: Practical Tips for Smoother AI Generation

{
    "ideas": [
        "Example first idea \n\n",
        ...
    ]
}

The Challenge: Latency

The hiccup? A waiting time of up to 30 seconds before users can view the AI’s suggestions. Watching a loading icon spin for half a minute is not a good user experience.

Tip #1: Leverage Streaming

export const createIdeas = async ({ occasion }: { occasion: string; }) => {
    return openai.chat.completions.create({
        model: chatModel,
        stream: true,
        messages: [
            {
                role: "user",
                content: `I want to host an event for the following occasion: ${occasion}. Write me a list of 4 separate ideas for this event`,
            },
        ],
    });
};

After understanding the mechanics, our next step was to synchronize the frontend client that communicates with the backend Express endpoint above with the frontend to support SSE.

export const requestIdeas = async (occasion: string) => {
    const res = await fetch(`${process.env.REACT_APP_API_BASE_URL}/generate-ideas`, {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json'
        },
        body: JSON.stringify({ occasion })
    });
    return res.body?.pipeThrough(new TextDecoderStream()).getReader();
};

The Impact

The results are night and day. From a staggering 30-second wait, users now see the initial AI-generated content within half a second.

Tip #2: UI Components of Streaming

When users are given multiple options to choose from, the UI should split those different options into different components. For example, each option should be a radio button or a different div. But streaming text in real-time throws a wrench in the works. How can we differentiate, parse each AI-generated suggestion as distinct UI components?

The Solution: Add parsing based on a unique character.

1. Add a unique Bullet Identifier: Update your prompt to the AI to use an unusual character as a bullet point that is unlikely to appear in the rest of your text. We used the “¶” symbol and updated our prompt to include the following: `Start each bullet point with a new line and the ¶ character. Do not include the normal bullet point character, the '-' character, or list numbers.``
2. Splitting the Stream: We segmented each byte array from the SSE endpoint into distinct words. This separation was pivotal, given that a single SSE byte array content was unpredictable. Sometimes it included a single word, other times it included full phrases that contained the “¶” character, like subject matter. ¶ Engage.
3. Append each word: Once each word is prepared, we append the value to the appropriate UI component. Tracking the “¶” occurrences helps us assign words to the correct component. For instance, a single “¶” means the content belonged to the first-option component. Repeating this process in the loop until the SSE endpoint closed.

export const parseResponseValue = (index: number, value: string, setters: Function[]) => {
    // Separate into individual words only (no phrases)
    const splitValue = value.split(/(?! {2})/g);

    for (let word of splitValue) {
        if (word.includes('¶')) {
            index++;
        } else {
            setters[index]((prev: string) => {
                return prev + word;
            });
        }
    }
    // Return the index to the calling function for use for the next byte array received from the endpoint. 
    return index;
};

export const requestIdeas = async (occasion: string) => {
    const res = await fetch(`${process.env.REACT_APP_API_BASE_URL}/generate-ideas`, {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json'
        },
        body: JSON.stringify({ occasion })
    });
    return res.body?.pipeThrough(new TextDecoderStream()).getReader();
};