Introduction
Imagination-Guided Open-Ended Text Generation, abbreviated as INLG, is a method that falls under the umbrella of natural language generation (NLG) that is used to generate open-ended text. The generation of text by INLG is guided by machine-generated image. INLG is a open-ended technique that enables creative and inventive text generation compared to other kinds of natural language generation (NLG), which require taking structured data as input and generating text that sufficiently and fluently conveys this data as output.
How does iNLG generate text
Text is generated through iNLG (Imagination-Guided Open-Ended Text Generation) utilizing a mixture of machine learning and human creativity. An outline of iNLG's text-generation process is as follows:
- Input: The input for iNLG consists of a series of images as well as a prompt.
- Encoding: Pre-trained image encoding is used to process the input images.
- Imagination: The encoded visuals are then utilized to direct a human writer in the generation of text. After considering the visuals and the prompts, the author utilizes their imagination to come up with a tale or descriptive passage.
- Machine Learning: The text that was generated is used to train a machine learning model, which learns to link the input images with the text that was generated.
- Text Generation: To generate new text from a series of input images and a prompt, a machine learning model must first be trained. The text is generated by the model by making a prediction about the words that are most likely to come next given the visuals and the prompt.
Overall, iNLG leverages the power of machine learning to produce inventive and cohesive open-ended texts by combining human ingenuity with the power of the imagination.
Implement iNLG using python
These are the steps you can take to implement iNLG with Python:
- Using the following command, you can get the iNLG repository from GitHub:
git clone git@github.com:VegB/iNLG.git
2. To organize your datasets, use the following commands:
cd iNLG/
for dataset in activitynet commongen rocstories
do
mkdir -p log/${dataset}
done
The above code:
- Transfers working directory to iNLG folder.
- Starts a loop that will go over each of the three datasets: activitynet, commongen, and rocstories.
- Creates a "log" directory within the current directory (iNLG), and then creates another directory inside that one with the name of the active dataset (activitynet, commongen, or rocstories). When you run mkdir with the "-p" option, it will create the parent directory if it doesn't already exist.
Note:
- The ROCStories dataset was developed to evaluate many aspects of machine learning models, including story understanding, story generation, and commonsense reasoning. It is often used in the challenge known as the story's Cloze Test, which requires models to correctly predict the end of the story that is provided to them.
- CommonGen dataset was created to evaluate a machine's ability to generate natural-sounding text, researchers have created the . It has approximately 35,000 distinct concepts described by over 79,000 commonsense descriptions. Crowdsourcing is used together with existing caption corpora to compile the dataset.
- We can read from the paper,
The ActivityNet dataset contains 200 different types of activities and a total of 849 hours of videos collected from YouTube. ActivityNet is the largest benchmark for temporal activity detection to date in terms of both the number of activity categories and number of videos, making the task particularly challenging.
3. Conda environment should be set up using the env.yml file that is supplied:
conda env create -f env.yml
conda activate inlg
python -m spacy download en
- Using the
conda env create -f env.ymlcommand will result in the creation of a new environment that includes all of the required dependencies. - You can make use of the spaCy library with English text to download the English language model with the command:
python -m spacy download en.
4. Using the below command, you can download the datasets and checkpoints:
python scripts/download_data_and_checkpoint.py
The above command is the path to the Python script that will be used to download the datasets and checkpoints. The iNLG implementation's script can be found in its "scripts" directory.
5. Use the following command to generate a text-only story:
bash scripts/run_story_generation_text_only.sh
- The script is executed through the bash command, and the scripts/run_story_generation.sh indicates the script's location.
- The run_story_generation_text_only.sh will generate a story through the iNLG (Imagination-Guided Open-Ended Text Generation) implementation.
6. The below command will generate an image-based story:
bash scripts/run_story_generation_with_image.sh
Completing these steps will enable you to produce open-ended text using Python and will set up the environment for iNLG. Take note that this implementation is derived on the iNLG paper "Visualize Before You Write: Imagination-Guided Open-Ended Text Generation," . So, your particular use case would call for extra adjustments or customization in order to accommodate it.
Conclusion
The results of the trials and analysis show that iNLG is successful at text completion, storytelling and concept-to-text generation. The application of INLG in natural language processing might lead to the creation of a flexible, interactive narrative-building tool. INLG is a promising technique in the study of natural language generation that might be used in many scenarios.
References
VegB
