Improving Controlled Text Generation via Neuron-Level Control Codes

Jay Orten, Nancy Fulda

2025

Abstract

Task-specific text generation is a highly desired feature for language models, as it allows the production of text completions that are either broadly or subtly aligned with specific objectives. By design, many neural networks switch between multiple behaviors during inference - for example, when selecting a target language in many-to-many translation systems. Such task-specific information is usually presented to the network as an augmentation of its input data. In this work, we explore an alternate approach: transmitting task information directly to each neuron in the network. This removes the need for task information to propagate forward during training, a particularly critical advantage in low-resource settings where maximum benefit must be extracted from each training example. To test this approach, we train over 160 language models from scratch with a large variety of architectures and configurations. Our results show that models with neuron-level augmentation can experience increased learning speed, improved final generation accuracy, and even novel learning capabilities, with greater benefits as network depth increases.

Paper Citation

in Harvard Style

Orten J. and Fulda N. (2025). Improving Controlled Text Generation via Neuron-Level Control Codes. In Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART; ISBN 978-989-758-737-5, SciTePress, pages 574-581. DOI: 10.5220/0013160100003890

in Bibtex Style

@conference{icaart25,
author={Jay Orten and Nancy Fulda},
title={Improving Controlled Text Generation via Neuron-Level Control Codes},
booktitle={Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART},
year={2025},
pages={574-581},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013160100003890},
isbn={978-989-758-737-5},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART
TI - Improving Controlled Text Generation via Neuron-Level Control Codes
SN - 978-989-758-737-5
AU - Orten J.
AU - Fulda N.
PY - 2025
SP - 574
EP - 581
DO - 10.5220/0013160100003890
PB - SciTePress