Scrape enchant descriptions

[wowsims]

Scrape enchant descriptions

[Kruhlmann]

I used the all_enchants.go file to scrape data from WOTLK Wowhead. Let me know if this is the kind of data you need.

There was an issue where DK runes and ring enchants were marked as items, while their ID's referred to spells, so I modified the following lines to include IsSpellID: true (fixed in #138):

{ID: 53343, EffectID: 3370, IsSpellID: true, Name: "Rune of Razorice", Phase: 1, Quality: proto.ItemQuality_ItemQualityUncommon, Bonus: stats.Stats{}, ItemType: proto.ItemType_ItemTypeWeapon, ClassAllowlist: []proto.Class{proto.Class_ClassDeathKnight}},
{ID: 53341, EffectID: 3369, IsSpellID: true, Name: "Rune of Cinderglacier", Phase: 1, Quality: proto.ItemQuality_ItemQualityUncommon, Bonus: stats.Stats{}, ItemType: proto.ItemType_ItemTypeWeapon, ClassAllowlist: []proto.Class{proto.Class_ClassDeathKnight}},
{ID: 53331, EffectID: 3366, IsSpellID: true, Name: "Rune of Lichbane", Phase: 1, Quality: proto.ItemQuality_ItemQualityUncommon, Bonus: stats.Stats{}, ItemType: proto.ItemType_ItemTypeWeapon, ClassAllowlist: []proto.Class{proto.Class_ClassDeathKnight}},
{ID: 54447, EffectID: 3595, IsSpellID: true, Name: "Rune of Spellbreaking", Phase: 1, Quality: proto.ItemQuality_ItemQualityUncommon, Bonus: stats.Stats{}, ItemType: proto.ItemType_ItemTypeWeapon, ClassAllowlist: []proto.Class{proto.Class_ClassDeathKnight}},
{ID: 54446, EffectID: 3594, IsSpellID: true, Name: "Rune of Swordbreaking", Phase: 1, Quality: proto.ItemQuality_ItemQualityUncommon, Bonus: stats.Stats{}, ItemType: proto.ItemType_ItemTypeWeapon, ClassAllowlist: []proto.Class{proto.Class_ClassDeathKnight}},
{ID: 53344, EffectID: 3368, IsSpellID: true, Name: "Rune of the Fallen Crusader", Phase: 1, Quality: proto.ItemQuality_ItemQualityUncommon, Bonus: stats.Stats{}, ItemType: proto.ItemType_ItemTypeWeapon, ClassAllowlist: []proto.Class{proto.Class_ClassDeathKnight}},
{ID: 70164, EffectID: 3883, IsSpellID: true, Name: "Rune of the Nerubian Carapace", Phase: 1, Quality: proto.ItemQuality_ItemQualityUncommon, Bonus: stats.Stats{}, ItemType: proto.ItemType_ItemTypeWeapon, ClassAllowlist: []proto.Class{proto.Class_ClassDeathKnight}},
{ID: 53342, EffectID: 3367, IsSpellID: true, Name: "Rune of Spellshattering", Phase: 1, Quality: proto.ItemQuality_ItemQualityUncommon, Bonus: stats.Stats{}, ItemType: proto.ItemType_ItemTypeWeapon, ClassAllowlist: []proto.Class{proto.Class_ClassDeathKnight}},
{ID: 53323, EffectID: 3365, IsSpellID: true, Name: "Rune of Swordshattering", Phase: 1, Quality: proto.ItemQuality_ItemQualityUncommon, Bonus: stats.Stats{}, ItemType: proto.ItemType_ItemTypeWeapon, ClassAllowlist: []proto.Class{proto.Class_ClassDeathKnight}},
{ID: 62158, EffectID: 3847, IsSpellID: true, Name: "Rune of the Stoneskin Gargoyle", Phase: 1, Quality: proto.ItemQuality_ItemQualityUncommon, Bonus: stats.Stats{}, ItemType: proto.ItemType_ItemTypeWeapon, ClassAllowlist: []proto.Class{proto.Class_ClassDeathKnight}},
{ID: 44645, EffectID: 3839, IsSpellID: true, Name: "Assault", Quality: proto.ItemQuality_ItemQualityCommon, Bonus: stats.Stats{stats.AttackPower: 40, stats.RangedAttackPower: 40}, ItemType: proto.ItemType_ItemTypeFinger, RequiredProfession: proto.Profession_Enchanting},
{ID: 44636, EffectID: 3840, IsSpellID: true, Name: "Greater Spellpower", Quality: proto.ItemQuality_ItemQualityCommon, Bonus: stats.Stats{stats.SpellPower: 23, stats.HealingPower: 23}, ItemType: proto.ItemType_ItemTypeFinger, RequiredProfession: proto.Profession_Enchanting},
{ID: 59636, EffectID: 3791, IsSpellID: true, Name: "Stamina", Quality: proto.ItemQuality_ItemQualityCommon, Bonus: stats.Stats{stats.Stamina: 30}, ItemType: proto.ItemType_ItemTypeFinger, RequiredProfession: proto.Profession_Enchanting},

Scraper output:

$ ./scrape <all_enchants.go | tee output
Item 38375: +12 Stamina
Item 38376: +18 Stamina
Item 44069: +50 Attack Power and +20 Resilience Rating
Item 44075: +29 Spell Power and +20 Resilience Rating
Item 44875: +30 Stamina and +25 Resilience Rating
Item 44137: +25 Frost Resistance and +30 Stamina
Item 44138: +25 Nature Resistance and +30 Stamina
Item 44139: +25 Shadow Resistance and +30 Stamina
Item 44140: +25 Arcane Resistance and +30 Stamina
Item 44141: +25 Fire Resistance and +30 Stamina
Item 44876: +30 Spell Power and 10 mana per 5 seconds.
Item 44877: +30 Spell Power and 20 Critical strike rating.
Item 44878: +37 Stamina and +20 Defense Rating
Item 44879: +50 Attack Power and +20 Critical Strike Rating
Spell 67839: Mind Amplification Dish
Item 44067: +40 Attack Power and +15 Resilience Rating
Item 44068: +23 Spell Power and +15 Resilience Rating
Item 44957: +30 Stamina and +15 Resilience Rating
Item 44129: +18 Spell Power and +10 Critical Strike Rating
Item 44130: +18 Spell Power and +5 Mana per 5 sec
Item 44131: +30 Attack Power and +10 Critical Strike Rating
Item 44132: +15 Dodge Rating and +10 Defense Rating
Item 44874: +24 Spell Power and +15 Critical Strike Rating
Item 44872: +24 Spell Power and +8 Mana per 5 sec
Item 44871: +40 Attack Power and +15 Crit Rating
Item 44873: +20 Dodge Rating and +15 Defense Rating
Spell 61120: +70 Spell Power and +15 Crit Rating
Spell 61118: +70 Spell Power and +8 Mana/5 seconds
Spell 61117: +120 Attack Power and +15 Crit Rating
Spell 61119: +60 Dodge Rating and +15 Defense Rating
Item 37330: +20 Arcane Resistance
Item 37331: +20 Fire Resistance
Spell 44483: +20 Frost Resistance
Spell 44494: +20 Nature Resistance
Spell 44590: +20 Shadow Resistance
Item 37347: +16 Defense Rating
Item 37349: Increased Stealth and +10 Agility
Item 44471: +225 Armor
Item 44472: +23 Haste Rating
Item 44488: +10 Spirit and 2% Reduced Threat
Spell 44582: +35 Spell Penetration
Spell 60609: +15 Haste Rating
Spell 44500: +16 Agility
Spell 60663: +22 Agility
Spell 55002: Flexweave Underlay
Spell 55642: Lightweave Embroidery
Spell 55769: Darkglow Embroidery
Spell 55777: Swordguard Embroidery
Item 37340: +20 Resilience Rating
Spell 44623: +8 All Stats
Item 44489: +10 All Stats
Spell 27958: +250 Mana
Spell 44492: +200 Health
Spell 47900: +275 Health
Spell 44509: +10 mana every 5 sec.
Spell 47766: +22 Defense Rating
Item 44484: +50 Attack Power
Item 44498: +30 Spell Power
Item 44944: +40 Stamina
Spell 44555: +16 Intellect
Spell 44593: +18 Spirit
Spell 44598: +15 Expertise Rating
Spell 44616: +6 All Stats
Spell 44635: +23 Spell Power
Spell 60616: +38 Attack Power
Spell 57683: +130 Attack Power
Spell 57690: +102 Stamina
Spell 57691: +76 Spell Power
Spell 57692: +70 Fire Resistance
Spell 57694: +70 Frost Resistance
Spell 57696: +70 Shadow Resistance
Spell 57699: +70 Nature Resistance
Spell 57701: +70 Arcane Resistance
Item 44485: +2% Threat and 10 Parry Rating
Spell 60668: +44 Attack Power
Spell 44592: +28 Spell Power
Spell 44484: +15 Expertise Rating
Spell 44506: Gatherer
Spell 44513: +35 Attack Power
Spell 44529: +20 Agility
Spell 44488: +20 Hit Rating
Spell 54998: Hand-Mounted Pyro Rocket
Spell 54999: Hyperspeed Accelerators
Spell 63770: +885 Armor
Item 38371: +45 Stamina and +15 Agility
Item 38372: +55 Attack Power and +15 Critical Strike Rating
Item 38373: +55 Stamina and +22 Agility
Item 38374: +75 Attack Power and +22 Critical Strike Rating
Item 44963: +40 Resilience Rating and +28 Stamina
Item 41601: +35 Spell Power and +12 Spirit
Item 41602: +50 Spell Power and +20 Spirit
Item 41603: +35 Spell Power and +20 Stamina
Item 41604: +50 Spell Power and +30 Stamina
Spell 60583: +55 Stamina and +22 Agility
Spell 60584: +75 Attack Power and +22 Critical Strike Rating
Spell 56034: +50 Spell Power and +30 Stamina
Spell 56039: +50 Spell Power and +20 Spirit
Item 44490: +32 Attack Power
Item 44491: +15 Stamina and Minor Speed Increase
Spell 60606: +24 Attack Power
Spell 44528: +22 Stamina
Spell 44508: +18 Spirit
Spell 44584: +7 Health and Mana every 5 sec
Spell 60623: +12 Hit Rating and +12 Critical Strike Rating
Spell 44589: +16 Agility
Spell 55016: +24 Critical Strike Rating
Spell 44633: +26 Agility
Spell 44510: +45 Spirit
Item 37339: Giantslaying
Item 37344: Deep Wounds
Item 41976: Titanium Weapon Chain
Item 44486: +65 Attack Power
Item 44487: +63 Spell Power
Item 44492: Berserking
Item 44494: Lifeward
Item 44495: Black Magic
Item 44496: +25 Hit Rating and +25 Critical Strike Rating
Spell 44629: +50 Spell Power
Spell 60621: +50 Attack Power
Spell 53342: Rune of Spellshattering
Spell 53323: Rune of Swordshattering
Spell 62158: Rune of the Stoneskin Gargoyle
Spell 44489: +20 Defense Rating
Spell 60653: +25 Intellect
Item 42500: Titanium Spike (45-67)
Item 44936: Titanium Plating
Spell 44645: +40 Attack Power
Spell 44636: +23 Spell Power
Spell 59636: +30 Stamina
Item 41146: +40 Ranged Haste Rating
Item 41167: +40 Ranged Critical Strike
Item 44739: Scope (+15 Damage)
Item 29186: +16 Defense Rating and +17 Dodge Rating
Item 29191: +22 Spell Power and +14 Hit Rating
Item 29192: +34 Attack Power and +16 Hit Rating
Item 30846: +17 Strength and +16 Intellect
Item 29193: +18 Stamina and +20 Resilience Rating
Item 19782: +10 Defense Rating +10 Stamina +30 Block Value
Item 28886: +18 Spell Power and +10 Critical Strike Rating
Item 28888: +30 Attack Power and +10 Critical Strike Rating
Item 28889: +15 Dodge Rating and +10 Defense Rating
Item 28909: +15 Critical Strike Rating and +12 Spell Power
Item 28910: +15 Critical Strike Rating and +20 Attack Power
Item 28911: +15 Defense Rating and +10 Dodge Rating
Item 20076: +18 Spell Power
Item 23545: +15 Spell Power and +14 Critical Strike Rating
Item 23548: +26 Attack Power and +14 Critical Strike Rating
Item 23549: +16 Stamina and +100 Armor
Item 33148: +12 Dodge Rating
Item 33150: 2% Reduced Threat
Item 11206: +3 Agility
Spell 34004: +12 Agility
Item 28274: +20 Spell Penetration
Item 28277: +15 Shadow Resistance
Item 35756: +12 Defense Rating
Spell 27957: +150 Health
Item 22546: +250 Mana
Item 24003: +6 All Stats
Item 28270: +15 Resilience Rating
Spell 33990: +15 Spirit
Spell 33991: +7 mana every 5 sec.
Item 35500: +16 Defense Rating
Item 22533: +12 Stamina
Item 22534: +15 Spell Power
Spell 34001: +12 Intellect
Spell 27899: +12 Strength
Spell 34002: +24 Attack Power
Spell 27905: +4 All Stats
Item 20727: +20 Shadow Spell Power
Item 28271: +15 Hit Rating
Item 28272: +20 Spell Power
Spell 33995: +15 Strength
Item 33152: +15 Agility
Item 33153: +2% Threat
Item 24274: +35 Spell Power and +20 Stamina
Item 24273: +25 Spell Power and +15 Stamina
Item 29533: +40 Attack Power and +10 Critical Strike Rating
Item 29535: +50 Attack Power and +12 Critical Strike Rating
Item 29536: +40 Stamina and +12 Agility
Item 16220: +5 Spirit
Item 35297: Minor Speed and +9 Stamina
Item 35298: +5 Health and Mana every 5 sec
Item 22543: +12 Stamina
Item 22544: +12 Agility
Item 28279: Minor Speed and +6 Agility
Item 22545: +10 Hit Rating and +10 Critical Strike Rating
Item 16250: +5 Weapon Damage
Item 22552: +7 Weapon Damage
Item 16252: Crusader
Item 22551: +30 Intellect
Item 22554: +70 Attack Power
Item 22555: +40 Spell Power
Item 22560: +50 Arcane and Fire Spell Power
Item 22561: +54 Shadow and Frost Spell Power
Item 22559: Mongoose
Item 19445: +15 Agility
Item 33165: +20 Agility
Item 22556: +35 Agility
Item 33307: Executioner
Item 35498: Deathfrost
Item 22539: +12 Intellect
Item 28282: +18 Stamina
Spell 44383: +12 Resilience Rating
Item 22535: +2 Weapon Damage
Item 22536: +12 Spell Power
Item 22538: +4 All Stats
Item 18283: +30 Ranged Hit Rating
Item 23765: Scope (+12 Damage)
Item 23766: Scope (+28 Critical Strike Rating)

Here's the scraper:

#!/usr/bin/env sh

set -e

spell_url_from_item_id() {
	safe_curl "https://www.wowhead.com/wotlk/item=$1" |
		grep -o '\/wotlk\/spell=[0-9]\+' |
		head -n 1
}

id_from_entry() {
	grep -oP '{ID: \K[0-9]+'
}

delete_newlines() {
	tr -d '\n'
}

safe_curl() {
	curl -sL "$1" |
		remove_user_posts |
		delete_newlines
}

spell_effect_from_url() {
	safe_curl "https://www.wowhead.com$1" |
		grep -oP '<th>Effect.*?</th>.*<a href="\/wotlk\/spell=[0-9]+/.*?">\K.*?(?=</a>)|<th>Effect.*?</th>.*<span class="q2">\K.*(?=</span>)'
}

remove_user_posts() {
	grep -v '<div class="user-post-">'
}

grep '^\s*{' |
	while read -r line; do
		if echo "$line" | grep 'IsSpellID: true' >/dev/null; then
			id=$(echo "$line" | id_from_entry)
			effect=$(spell_effect_from_url "/wotlk/spell=$id")
			echo "Spell $id: $effect"
		else
			id=$(echo "$line" | id_from_entry)
			spell_url_suffix=$(spell_url_from_item_id "$id")
			effect=$(spell_effect_from_url "$spell_url_suffix")
			echo "Item $id: $effect"
		fi
		sleep 2
	done

[lologarithm]

awesome work, @Kruhlmann, are you in our development discord?

[Kruhlmann]

Yes, as ges#9680

[jimmyt857]

Nice! I think the output should be in the form ${effectID}: "${description}" and stored in assets so we can use it in the UI. The scraper should also be committed in case we have to modify/add/remove enchants in the future.

I can do this myself if necessary, but @Kruhlmann you should get the credit so go for it!