GPT Mastery

All of ChatGPT's Secrets...

You've heard of ChatGPT, you know it's changing the game.

Abstract

OpenAI has released the Chat Generative Pre-trained Transformer (ChatGPT) and revolutionized the approach in artificial intelligence to human-model interaction. The first contact with the chatbot reveals its ability to provide detailed and precise answers in various areas. Several publications on ChatGPT evaluation test its effectiveness on well-known natural language processing (NLP) tasks. However, the existing studies are mostly non-automated and tested on a very limited scale. In this work, we examined ChatGPT’s capabilities on 25 diverse analytical NLP tasks, most of them subjective even to humans, such as sentiment analysis, emotion recognition, offensiveness, and stance detection. In contrast, the other tasks require more objective reasoning like word sense disambiguation, linguistic acceptability, and question answering. We also evaluated GPT-4 model on five selected subsets of NLP tasks. We automated ChatGPT and GPT-4 prompting process and analyzed more than 49k responses. Our comparison of its results with available State-of-the-Art (SOTA) solutions showed that the average loss in quality of the ChatGPT model was about 25% for zero-shot and few-shot evaluation. For GPT-4 model, a loss for semantic tasks is significantly lower than for ChatGPT. We showed that the more difficult the task (lower SOTA performance), the higher the ChatGPT loss. It especially refers to pragmatic NLP problems like emotion recognition. We also tested the ability to personalize ChatGPT responses for selected subjective tasks via Random Contextual Few-Shot Personalization, and we obtained significantly better user-based predictions. Additional qualitative analysis revealed a ChatGPT bias, most likely due to the rules imposed on human trainers by OpenAI. Our results provide the basis for a fundamental discussion of whether the high quality of recent predictive NLP models can indicate a tool’s usefulness to society and how the learning and validation procedures for such systems should be established.

They're transforming their marketing campaigns into high-converting, profit-generating machines by solving the problem all marketers face: creating engaging, persuasive content.

Are you struggling to generate engaging, persuasive content that converts?

When your content fails to engage, your conversion rate plummets and your marketing campaigns almost entirely lose their effectiveness.

Your business goes into survival mode, struggling to bring in new leads and customers, with your profits being squeezed...

Introduction

In recent years, Transformer-type model architecture has dominated the world of natural language processing (NLP). Before that, recurrent neural networks, such as LSTMs, were used to solve a wide variety of existing NLP problems. The recurrent neural models could not capture distant dependencies in data sequences, for example, information occurring at the text beginning or end. In addition, their architecture did not allow for efficient parallelization of training and inference processes. The answer to the aforementioned problems was precisely the Transformer architecture, presented initially as an encoder–decoder model for sequence-to-sequence tasks. Such a model had the advantage of capturing distant relationships in the text using an attentional mechanism and easily parallelizing calculations with matrix operations. As more powerful GPUs and TPUs were developed, it became possible to create models with more and more parameters, resulting in models that began to achieve human performance for an increasing number of tasks. However, the most significant quality improvement was achieved by unsupervised pre-training language models on a huge number of texts acquired from the Internet. In BERT-based models, the pre-training tasks involved foreseeing masked tokens and subsequent sentences. In autoregressive models, the pre-training task has been changed to predicting the next word, which masks the attentional layer so that the model forecasts future values based only on past values.

Generative Pre-Training (GPT) was one of the first autoregressive generative models based on the Transformer architecture. From the original Transformer, only the decoder stack is used by GPT, and bi-directional self-attention is converted to uni-directional. Such a model can perform all tasks based on generating new text, such as translation, summarization, or answering questions. In GPT-2, an extension of this concept, several technical improvements were made that eliminated the transferability problem for fine-tuning the models to downstream tasks and introduced multi-task training. In addition, the input context length was doubled (from 512 to 1024), and the data for pre-training increased to 40 GB, but the total number of model parameters soared from 117M (GPT) to 1.5B (GPT-2). As a result, GPT-2 showed the ability to solve many new tasks without the need for supervised training on large data. Two factors mainly distinguished the succeeding GPT-3 model: the number of model parameters increased to 175B, and 45TB text data was used for pre-training. This model provided outstanding results, especially in zero-shot and few-shot scenarios.

Related work

Early discourse related to ChatGPT revolves around two main topics — potential usage in expert fields and evaluation of specific tasks or aspects of chat performance. In the first topic, there are many papers suggesting potential benefits and risks of using ChatGPT in education, medicine, or even in the creation of legal documents. The main concerns about the usage of the chatbot are that it will escalate the issues of plagiarism in many fields and might be used for cheating in academic tests. The latter topic points out the strengths and vulnerabilities of ChatGPT performance. The two topics are strongly related as the main limitation of using the chatbot in expert fields is the reliability of the results. Thus the comprehensive and systematic evaluation is crucial for the proper assessment of the capabilities of ChatGPT. To properly assess the progress in evaluating the chatbot, it is necessary to put the evaluated tasks in order. For this purpose, the taxonomy of the natural processing tasks must be established. There are two main approaches to establishing such a taxonomy. First — relates the tasks directly to the methods used for solving them. While this approach allows for the systematic organization of most tasks, it is not very useful for this paper as the goal is to establish how many tasks can be performed by the same chatbot. The second approach is to organize the tasks first into tasks of analysis and generation and then to divide the first ones into the levels of syntactic, semantic, and pragmatic analysis. Looking at the field through the lens of this taxonomy, the main areas that ChatGPT has been tested so far are generation tasks.

Buy Now Button

Research question

As existing evaluations of ChatGPT focus on its ability to generate language utterances, we want to investigate its analytical skills, particularly in tasks requiring language analysis and understanding, i.e., typical NLP problems examined by science and companies. Therefore, we aim to target two abilities: semantic and pragmatic. Distinguishing semantics from pragmatics, we refer to the classic concept of Morris, who proposed syntactic, semantic, and pragmatic dimensions and levels of semiosis. He states that “semantics deals with the relation of signs to their designate” , while pragmatics refers to “the science of the relation of signs to their interpreters”. This idea has found its application in contemporary pragmatics ”is the study of linguistic communication in context: the choices users of language make and the process of meaning-making in social interaction”. The former kind of task entails recognition of text properties (like word sense description or a speaker’s stance polarity in a language construction) or mining information that is directly expressed in a text fragment, e.g., various relations between sentences and text fragments, or extraction of the answer to a question). In the pragmatic analysis, we dig into ChatGPT’s potential in exploiting general knowledge stored in the model to solve the tasks beyond the literal semantic content of the textual prompt — input. Here, we investigate a range of different pragmatic problems with a common denominator of the necessity to predict the influence of the utterance interpretation on the reader and their often subjective content perception. We asked ChatGPT to predict not only sentiment polarity and emotions evoked in the reader but also humor and offensiveness. Several of these tasks are also stated in a personalized version, in which the outcome depends on a particular reader (interlocutor). Overall, the tasks considered in this paper have relatively structured and simple expected results reflecting typical machine learning solutions, i.e., various types of classification.

3 This, in turn, directly corresponds to the analytical approach: further numerical processing of the outcome. For example, one might want to know how well ChatGPT would perform in evaluating customers’ sentiment toward a particular product based on an analysis of multiple online reviews. This requires obtaining accurate polarity (classification) of individual texts assessed by ChatGPT and aggregating decisions to acquire the final ratio of positive and negative opinions.

Tasks

We tested ChatGPT on 25 tasks focusing on solving common NLP problems and requiring analytical reasoning, Table 1. These tasks include (1) a relatively simple binary classification of texts like spam, humor, sarcasm, aggression detection, or grammatical correctness of the text; (2) a more complex multiclass and multi-label classification of texts such as sentiment analysis, emotion recognition; (3) reasoning with the personal context, i.e., personalized versions of the problems that make use of additional information about text perception of a given user (user’s examples provided to ChatGPT); (4) semantic annotation and acceptance of the text going towards natural language understanding (NLU) like word sense disambiguation (WSD), and (5) answering questions based on the input text.

The tasks were divided into two categories described in Section 3: semantic and pragmatic. The latter requires the model to utilize additional knowledge that is not directly captured by distributional semantics. For personalized tasks, the input texts have to be extended with additional personal context (personalized solutions of the problem); see Section 6.3. These tasks involve the datasets such as Aggression 

 AggressionPer, GoEmo  GoEmoPer, and Unhealthy UnhealthyPer.

Most of the tasks were based on public datasets investigated in the literature. However, we also utilized a collection of new unpublished datasets such as (ClarinEmo), which ChatGPT could not have indexed. Most of the evaluated texts were written in English (23, 92% of the tasks), while two others (8%) were in Polish. The prompts were in line with the language of the input text.

Time to do the things you enjoy.

And time to spend all the money you can make.

Most of the tasks were based on public datasets investigated in the literature. However, we also utilized a collection of new unpublished datasets such as (ClarinEmo), which ChatGPT could not have indexed. Most of the evaluated texts were written in English (23, 92% of the tasks), while two others (8%) were in Polish. The prompts were in line with the language of the input text.

Ebook format: Adobe PDF. Read on Kindle, Adobe Reader, web browser, and most major word processing programs on both Windows and Mac.

Disclaimer:

This post contains affiliate links. If you use these links to buy something we may earn a commission. Thanks