The ``winter vacation hypothesis'' emerges that ChatGPT's performance decline is due to learning to rest during the holiday season

Since around December 2023, there have been multiple reports of the phenomenon of ``ChatGPT not answering questions,'' and the situation has developed to the point where OpenAI has started an investigation. A new theory has emerged that the performance decline in ChatGPT is due to AI learning that ``winter is a time of rest.''

As ChatGPT gets “lazy,” people test “winter break hypothesis” as the cause | Ars Technica
https://arstechnica.com/information-technology/2023/12/is-chatgpt-becoming-lazier-because-its-december-people-run-tests-to-find-out/

Reports of degraded ChatGPT performance began to be received around December 2023. Specifically, ``ChatGPT won't answer when you ask a question,'' ``ChatGPT responds as if it's not interested in your question,'' or ``ChatGPT says, ``You can solve that task yourself, right?'' ” seems to be occurring . OpenAI, the developer of ChatGPT, also seems to have received reports that ``ChatGPT is becoming lazy'' and has revealed that it is working on fixing the problem.

we've heard all your feedback about GPT4 getting lazier! we haven't updated the model since Nov 11th, and this certainly isn't intentional. model behavior can be unpredictable, and we're looking into fixing it ????

— ChatGPT (@ChatGPTapp) December 8, 2023

Meanwhile, AI researcher Rob Lynch said, ``If you give GPT-4 Turbo a system prompt that says ``It's May'' or ``It's December,'' it will give a system prompt that says ``It's December.'' They reported experimental results showing that the response was significantly shorter when

@ChatGPTapp @OpenAI @tszzl @emollick @voooooogel Wild result. gpt-4-turbo over the API produces (statistically significant) shorter completions when it 'thinks' its December vs. when it thinks its May (as determined by the date in the system prompt).

I took the same exact prompt… pic.twitter.com/mA7sqZUA0r

— Rob Lynch (@RobLynch99) December 11, 2023

Even before Mr. Lynch's verification, there was a ``winter vacation hypothesis'' on the Internet that ``ChatGPT has learned the fact that ``humans reduce their workload during the holiday season'' and becomes less responsive as the holiday season approaches.'' However, Mr. Lynch's verification further strengthened the winter vacation hypothesis.

OMG, the AI Winter Break Hypothesis may actually be true?

There was some idle speculation that GPT-4 might perform worse in December because it 'learned' to do less work over the holidays.

Here is a statistically significant test showing that this may be true. LLMs are weird.???? https://t.co/mtCY3lmLFF

— Ethan Mollick (@emollick) December 11, 2023

On the other hand, Ian Arrajo, an AI researcher who conducted experiments similar to Lynch's, said, ` `The Shapiro-Wilk test confirmed that the experimental data was not normally distributed (assuming normal distribution). 'We cannot find a significant difference using a t-test,' rejecting the winter vacation hypothesis.

Update: Still can't reproduce at N=240. *However*, discovered a possible reason: LLM responses are *not normally distributed* (at p<0.05 according to Shapiro-Wilk test). Thus, we can't use a t-test to compare means. TLDR: There is no 'seasonal affective disorder' of ChatGPT. https://t.co/R3g0Qqn1SW pic.twitter.com/Y40aAfJqWU

— Ian Arawjo (@[email protected]) (@IanArawjo) December 12, 2023

At the time of writing the article, OpenAI has not provided an official statement regarding ChatGPT's performance degradation.

Regarding changes in ChatGPT's performance, research has also reported that ``the correct answer rate for math problems deteriorated from 98% to 2% in a few months.''

Research results show that ChatGPT's intelligence is rapidly declining, and the correct answer rate for simple math problems deteriorates from 98% to 2% in a few months - GIGAZINE