franec94/pivot_analysis

The pivot analysis algorithm for lexical analysis of text attribute transfer

The Pivot Analysis

The implementation of paper Yao fu, Hao Zhou, Jiaze Chen, and Lei Li, Rethinking Text Attribute Transfer: A Lexical Analysis. INLG 2019 (oral). link

In this paper, we discuss the observation that in many text style transfer datasets and models, only a few style-related words are changed during the transfer process, while the higher-level sentence structures remain unchanged. E.g. to change a negetive sentence "The food is awful" in Yelp to positive, one only need to substitute the word "awful" -> "The food is awesome".

How can quantitatively identify, measure, and visualize the influence of these words? We propose three algorithms for this purpose: the pivot word discovery, the pivot classifier, and the precision-recall histogram algorithms. They are all implemented in this repo.

We gather 8 major style-transfer dataset, standarlize them (so in your future work you could use them from this repo with minimal modification :), and analyze the pivot effects in these dataset. All analytical results from the paper can be reproduced and find out in the outputs/ folder.

Download the data

The datasets used in the paper are:

• yelp
• amazon
• caption
• gender
• paper
• politics
• reddit
• twitter

All organized as: train.0, train.1/ dev.0, dev.1/ test.0, test.1. Download from here

But note that the caption dataset does not have the right test data (because they made a mistake in their release, the positive and negative sentences in the test set are the same).

Other data are from the corresponding papers, with renaming and re-organization to fit our code.

Run it

mkdir outputs
python main.py --dataset=yelp --pivot_thres_cnt=1 --prec_thres=0.5 --recl_thres=0.0

and the outputs would something like:

...
Pivot word discovery:
class 0, 4929 pivots, pivot recall: 0.3348
class 1, 4129 pivots, pivot recall: 0.3435
...
Pivot classifier:
train accuracy: 0.8401
dev accuracy: 0.8313
test accuracy: 0.8333
...
output stored in
../outputs/yelp_1.pivot

Sample outputs

yelp_0.pivot: word/ precision/ recall (negative sentiment)
sadly			0.9924	0.0002
mistaken		0.7778	0.0000
general			0.6285	0.0001
run			0.6795	0.0003
mill			0.6226	0.0000

yelp_1.pivot: word/ precision/ recall (positive sentiment)
hoagies			0.7903	0.0000
italian			0.7029	0.0004
ton			0.7260	0.0001
really			0.5998	0.0038
worthy			0.6548	0.0000

yelp_0.sent: (pivot words are annotated with their precision)
ok(0.927) never(0.897) going(0.680) back(0.616) to this place again .
easter(0.786) day(0.502) nothing(0.918) open(0.516) , heard(0.778) about this place figured(0.781) it would ok(0.927) .

yelp_1.sent: (pivot words are annotated with their precision)
staff(0.791) behind the deli(0.696) counter were super(0.845) nice(0.907) and efficient(0.943) !
the staff(0.791) are always(0.918) very nice(0.907) and helpful(0.890) .

Parameters tunning:

prec_thres gives the confidence of how a word may determine the classification. To find strong pivot words, increase this parameter (e.g. [0.7, 1.0]). To achieve better classification performance, decrease this parameter (e.g. [0.5, 0.7])

recl_thres and pivot_thres_cnt prevents overfitting on single words. To increase confidence of the pivot words, increase them; to increase classification performance, decrease them.

Contact

Yao Fu, [email protected]