pyfa-org/eos

Eos - library for modeling EVE online ship fits

eos's Issues

Overload/heat implementation

Theory is here: https://forums.eveonline.com/default.aspx?g=posts&t=32225

Going to back-up it here too:

Heat gain law:
H(t) = heatCapacity / 100 - e ^ (-t * heatGenerationMultiplier * sum(heatAbsorbtionRateModifier)), where:
t - time in seconds (or we can call it 'tick', as TD will make actual server seconds slower) since you started overheating your mods, at which rack temperature should be 0
H(t) - temperature of rack (absolute) at time t
heatCapacity - value of attribute ['heatCapacityHi' | 'heatCapacityMed' | 'heatCapacityLow'] of your ship
heatGenerationMultiplier - value of attribute 'heatGenerationMultiplier' of your ship
sum(heatAbsorbtionRateModifier) - sum of values of 'heatAbsorbtionRateModifier' attributes of all modules you're overheating in given rack

Heat dissipation law:
H(t) = H(0) * e ^ (-t * heatDissipationRate), where:
t - time in ticks since you started observation
H(t) - temperature of rack (absolute) at time t
H(0) - temperature of rack (absolute) when you started observation
heatDissipationRate - value of attribute ['heatDissipationRateHi' | 'heatDissipationRateMed' | 'heatDissipationRateLow'] of your ship

Notes:
Heat gain/dissipation in any given rack is completely isolated. You can overload your MWD while having high slot rack being cooled.
Rack heat dissipation stops when you overload any module in given rack.

Heat damage
The easiest stuff: when module is overloaded, it damages itself and other mods by the value of its 'heatDamage' attribute (modified by your skills, ship and any other possible modification sources).

Chance to damage
When you overload any module, it has chances to damage any module from its rack
P = Fh * Fs * Fa, where:
P - chance to damage given module (absolute)
Fh - heat factor, described in details below
Fs - slot factor, described in details below
Fa - attenuation factor, described in details below

Fh = H, where:
H - heat of given rack (absolute) by the end of overheated module cycle

Fs = (Ho + Mo + Lo) / (Hs + Ms + Ls + Rs), where:
Ho - number of online+ modules in high rack
Mo - number of online+ modules in medium rack
Lo - number of online+ modules in low rack
Hs - total number of slots in high rack
Ms - total number of slots in medium rack
Ls - total number of slots in low rack
Rs - total number of rig slots
'online+' term includes all online, active and overloaded modules. That is, offlined modules act as empty slot here.

Fa = heatAttenuation ^ distance, where
heatAttenuation - value of attribute ['heatAttenuationHi' | 'heatAttenuationMed' | 'heatAttenuationLow'] of your ship
distance - distance of given module from overloaded module (that is, overloaded module itself has distance 0), rack isn't looped - so first and last modules are not adjacent

Notes:
Module is assigned to some rack and position within this rack based on its real position, visual rearrangement many players do in space has no effect on this
If overloaded module is killed mid-cycle by other overloaded module, it instantly goes offline - stops affecting ship, and doesn't damage other mods (both at the moment of death and at the expected end of cycle)

Speculations:
Reliance on 'heatCapacity' series of attributes: i wasn't able to check it, as all of the ships have it at 100. It may be possible that its other value doesn't affect anything at all - could be hardcoded
Reliance on 'heatDissipationRate' series of attributes: i wasn't able to check it, as all of the ships have it at 0.01. It may be possible that its other value doesn't affect anything at all - could be hardcoded

Tests in dist package

from pip.req import parse_requirements messes with find_packages(exclude=['tests']) somehow, disabling its filtering capability. Without this import, it filters out tests as expected.

Add support for t3 destroyers

List of things to do (these destroyers use different effect scheme under the hood):

• Write single modifier generator which will decide if modifierInfo or expression tree should be used (current modifierGenerator should be used as one of its modules)
• Write modifierInfo-to-modifiers converter
• Fix existing modifierGenerator tests and write new ones for modifierInfo
• In cache generator, make sure to take into consideration references from modifierInfo to skills, groups and attributes during cleanup stage and test it
• Add separate mode attribute to fit
• Check if anything needs to be changed in restriction tracker
• Find out how destroyers are connected with their modes

Eos API improvements

Rewrite top-level API to be more like what pyfa-experimental has:

• Source manager
• Ability to specify source for fits (like we currently specify eos)

This will also let us to improve logging - log not only when fit is assigned to Eos instance, but always (use python-wide logger which is actually possible now, but less motivation to do it).

Also, investigate if it's possible to consistently fail various operations on fit / holders when no source assigned to fit (ATM result is barely predictable, let alone consistent) - by returning empty values (None/()/{} etc) or raising special exception. If possible, it should cost nothing during regular operation (when fit has source) but it's probably okay to sacrifice some performance when switching fit's source.

Skills Require Sub-requirements Incorrectly

Posted by @Ebag333

Simple example. Take a Rig (which has other issues with skills, but that's another issue, literally).

Basic rigs skill requirement looks something like this:

Jury Rigging I
       Mechanics III

In order to train Jury Rigging I, you need to have Mechanics III.

But what if a character did have Jury Rigging I and somehow never trained Mechanics III? Okay, this isn't possible for rigs but it is possible for a number of ships/modules that CCP has reworked, the most prominent being Command Ships. There are many pilots who can fly a command ship, but cannot run links (soon to be bursts), and links are a prerequisite to command ships.

For those pilots, Eos will currently throw an exception, when it's actually a completely legal fit.

Take this code:

from eos import *
from eos.holder_filter import *

data_handler = JsonDataHandler('C:\Temp\PhobosJSONDump\TQ')  # Folder with Phobos data dump
cache_handler = JsonCacheHandler('C:\Temp\PhobosJSONDump\EosCache\eos_tq.json.bz2')
SourceManager.add('tiamat', data_handler, cache_handler, make_default=True)

fit = Fit()

#fit.skills.add(Skill(26252, level=1)) # Jury Rigging I
#fit.skills.add(Skill(3392, level=3)) # Mechanics III

fit.ship = Ship(40340)  # Upwell Palantine Keepstar
fit.rigs.equip(Rig(37275))  # Standup XL-Set Extinction Level Weapons Suite II

fit.validate()
pass

As it stands, this code will throw an exception that you're missing skill 26252, as it should.

Uncomment the line with 26252. Eos will throw an exception that you are missing skill 3392. Which it should not.

Uncomment both lines and Eos is happy.

This should be a fairly straight forward (hopefully) change to make the validation not recursive on ships/modules/rigs/etc.

Fighters

Posted by @Ebag333

Slots for fighters don't seem to exist at all in Eos.

We could treat them as drones, but that might get a bit complicated as fighters have multiple fighters per flight (example: 4 out of a maximum of 9), while drones are always a 1/1 ratio.

It's probably best to add a new spot for them.

fit.fighter.add(Fighter(2948, state=State.active, count=4))

An example fighter from the JSON:

  "2948": {
    "descriptionID": 93600, 
    "typeID": 2948, 
    "graphicID": 10039, 
    "capacity": 0.0, 
    "description": "<font color=yellow>Subject:</font> Prototype Nation Vessel (ID:Shadow)<br>\r\n<font color=yellow>Military Specifications:</font> Heavy Fighter class vessel. Moderate microwarp velocity. Enhanced weapons systems. Reverse-engineering has been made possible through salvage of specific Sansha's Nation vessels.<br>\r\n<font color=yellow>Additional Intelligence:</font> At this initial stage conclusions are hard to form, but early indications from the recovery of three individual bomber pilots – all of them heavily augmented – strongly indicates that these crew members are the same planetary abductees found in other Nation prototype vessels. Cross-referencing recovered DNA with missing persons reports has so far revealed little, though even at this stage the origin of these pilots is in little doubt.<br>\r\n<i>Synopsis from ISHAEKA-0107. DED Special Operations.</i><br><i>Authorized for Capsuleer dissemination.</i>   \r\n", 
    "typeNameID": 105075, 
    "published": true, 
    "marketGroupID": 1310, 
    "volume": 1800, 
    "typeName": "Shadow", 
    "radius": 35, 
    "basePrice": 20336332, 
    "mass": 2000, 
    "groupID": 1653, 
    "portionSize": 1
  }, 

Structures Not Supported by Eos

Posted by @Ebag333

Eos does not like citadels.

Keepstar:
fit.ship = Ship(35834)

Since Citadels are essentially ships in game, and use the same fitting tool, can we just treat them like ships? Seems like that path of least resistance to me.

Connected to pydev debugger (build 162.1967.10)
Traceback (most recent call last):
  File "C:\Python34\lib\site-packages\eos\data\cache_handler\json_cache_handler.py", line 89, in get_type
    type_ = self.__type_obj_cache[type_id]
  File "C:\Python34\lib\weakref.py", line 131, in __getitem__
    o = self.data[key]()
KeyError: 35834

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python34\lib\site-packages\eos\data\cache_handler\json_cache_handler.py", line 95, in get_type
    type_data = self.__type_data_cache[json_type_id]
KeyError: '35834'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2016.2.3\helpers\pydev\pydevd.py", line 1580, in <module>
    globals = debugger.run(setup['file'], None, None, is_module)
  File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2016.2.3\helpers\pydev\pydevd.py", line 964, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2016.2.3\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Users/Ebag333/Documents/GitHub/pyfa-ng/pyfa.py", line 17, in <module>
    fit.ship = Ship(35834)  # Navy Typhoon
  File "C:\Python34\lib\site-packages\eos\fit\holder\container\single_onfit.py", line 57, in __set__
    instance._add_holder(new_holder)
  File "C:\Python34\lib\site-packages\eos\fit\fit.py", line 113, in _add_holder
    holder._fit = self
  File "C:\Python34\lib\site-packages\eos\fit\holder\mixin\holder\holder.py", line 59, in _fit
    self._refresh_source()
  File "C:\Python34\lib\site-packages\eos\fit\holder\mixin\holder\holder.py", line 78, in _refresh_source
    self.item = type_getter(self._type_id)
  File "C:\Python34\lib\site-packages\eos\data\cache_handler\json_cache_handler.py", line 97, in get_type
    raise TypeFetchError(type_id) from e
eos.data.cache_handler.exception.TypeFetchError: 35834

Boosters

Need to implement:

• ways to detect effects corresponding to booster side-effects
• expose some kind of info and control API over side-effects
• add support for togglable effects in attribute calculator

Capital Size Validation

Posted by @Ebag333

Making this a separate issue to avoid contaminating #18 anymore.

From Kadesh:

It's the way it was done in EVE. I added this verification after i was playing with abaddon fit on SiSi and noticed that EVE doesn't let me to do that. The limitations in game client were exactly the same as implemented in this file: https://github.com/pyfa-org/eos/blob/master/eos/fit/restriction_tracker/register/capital_item.py I repeat, it is not a hack or my assumption, i took this data from reaction of server to my actions and from error message which client shown me.

if you find any discrepancies between ingame stuff and eos implementation, file new issue. Current CCP implementation might be different indeed (something based on ship's base PG and turret base pg requirement, for example - to prevent not just capital guns being fit on subcaps, but also large guns being fit onto frigs).

To be fair, I haven't hacked the client to see how exactly they limit this, but here's what I've seen (and this was happening on TQ, not SiSi).

Everyone knows that you can't fit dread guns on sub-caps, but then CCP went and added a ton of modules for caps (so they could stop using battleship sized stuff). We're talking afterburners, shield extenders, armor plates, the works. They went nuts.

Hilarity ensued.

You had 500k EHP capital LSE shield Bhaals running around (with insane regen), someone put stuff on the mega hull (can't remember if it was a Mega or one of the variants), basic chaos.

CCP blocked that, but missed a few modules. They they later fixed the modules they missed (none of those modules were particularly useful anyway).

So I believe (and very well could be wrong) that CCP has a hand done list somewhere that blocks capital mods.

Anyway. All that to say.

There's a couple issues with this validation in general:

• It assumes all modules over 500 volume are capital modules. This is not true.
• It assumes that all modules under 500 volume are not capital modules. This may or may not be true (but even if it's true today, there's nothing guaranteeing us that it will be true next patch).
• It assumes that all "capital modules" have a skill with a capital type. This is not true.

Okay, so to break this down.

It assumes all modules over 500 volume are capital modules.

So we have the code which checks if:
MAX_SUBCAP_VOLUME = 500

It is true that capital modules are very large in volume (doing a search in the database, they mostly seem to be exactly 4000 actually, but a lot of them also have special volumes).

The problem isn't so much with saying that "capital modules are over 500" as it is saying "all modules over 500 are capital modules."

There are many things in game that aren't a capital module that are over 500 in volume. The big one that we'll care about is structure modules. Now, granted, we can check if it's a structure module or not fairly easily, but coding in a one off exception to the rule is something that we should handle with care.

On the flip side, we assume that anything under 500 volume is not a capital module, when this actually isn't true.

Unit W-634's Modified Fighter Support Unit is a carrier module (so a capital module), yet it only has a volume of 200, and the capital module validation check would not apply (even though it should).

So validating things larger than 500 is problematic, because of the modules in game that are large yet aren't capital.

Validating things that are less than 500 is problematic, because of the couple capital modules that are very small.

We can't lower that threshold, because if we set it to 200 (to grab Unit W-634's Modified Fighter Support Unit) we'll grab subcap modules (like Bastion Module I).

So the TL;DR is....filtering based on size is problematic. :(

A further check is done (and this one actually isn't called out in the exception that gets returned) on the skill.

if (Type.capital_ships in ship_item.required_skills):

The problem is that many of the capital items added in game don't require any special skills to use. For example, Capital Shield Extender II has the exact same requirements as Large Shield Extender II. So this check will incorrectly not validate these modules as capital modules, when it should.

Unfortunately I don't have a magic bullet that will fix this. This is a really difficult thing to tackle because there's no single clear cut value that identifies these modules as capital or not.

Most likely our best solution is probably to maintain a list of groupID and/or marketGroupID to filter on. I don't like it, as it's hard coding stuff, but I don't see many other options.

Logger improvements

Consider logger changes where it will write log for each day to separate file (and remove date field from contents of file, and make it to take path to log folder instead of path to log file).

expressions-to-modifiers converter improvements

Investigate if it's possible to simplify it by getting rid of shared.operand_data map and using modifierInfo-to-modifier-like approach (simpler version of filter_map)

AAR support

AAR uses custom effect to multiply its rep power. We need support for custom effects in Eos attribute calculator.

Reactive Armor Hardener (AKA Adaptive) Implementation

Posted by @Ebag333

Reactive Armor Hardener is something that's difficult to handle, but because we are abstracting the fitting engine from everything else, it makes it easy (well, easier at least).

So first off, how the RAH works. (This has been covered before in about 10 million pages of detail, but just a quick summary).

1. Each cycle the RAH sorts inoming damage actually taken (this is post resist, or what shows in your logs) first by damage amount, secondarily alphabetically (to break potential ties).
2. If the two resists that took the least damage still have resists assign to them, it will steal that (up to 3% per resist).
3. It will then divide the amount stolen, and apply it (equally) to the two resists that otok the most damage.

There are a couple of caveats:

• It will never take resists from the two resist types that took the most damage
• In a caveat to the caveat, if the second highest resist does not take any damage (0 damage), the it will steal resists from there (same rules/process applies as listed above).

So what does this mean for Eos? Well, because we're abstracting it out, we don't care about processes over time. Eos is a snapshot, not a movie. So for Eos, we only need to do a few things.

1. Accept an incoming damage profile.
2. Accept the RAH module with a profile.
3. Calculate resists according to the above data.
4. Return the fit results (which we do anyway) and (this is important!) the damage profile post resists.

Eos doesn't adjust the RAH in any way, or run any calcs as to how it works. All it's doing is calculating how everything looks with the current stats, so there's no simulation done on Eos's part. Part of the reason we don't want to do that is because of the stacking penalty between RAH and DCU, if we try and simulate it inside Eos (as we do within Pyfa now), we can't properly calculate the stacking penalty (which throws us off).

So how do we handle the simulation? With the framework/skeleton.

So here's some psuedocode as to how it might look. I'm not sure if damage profiles are handled yet, and I'm sure that Eos won't return actual damage taken.

This is essentially code from the framework calling Eos. Doing it this way means we just have a single loop an literally a handful of lines of code, and we can use this method to simulate anything (heat, capacitor over time, etc).

fit = Fit()

fit.ship = Ship(587)  # Rifter
fit.modules.low.equip(ModuleLow(123, state=State.active))  # (This isn't the actual RAH ID, but meh)

profiles.add(Damage(EM, Amount=0)
profiles.add(Damage(Kin, Amount=100)
profiles.add(Damage(Therm, Amount=100)
profiles.add(Damage(Exp, Amount=0)

profiles.add(Adaptive(EM, Amount=.15)
profiles.add(Adaptive(Kin, Amount=.15)
profiles.add(Adaptive(Therm, Amount=.15)
profiles.add(Adaptive(Exp, Amount=.15)


while Loop=1:
    fit.validate()

    # This simply sees if we're in a loop or done, and kicks us out of the loop
    CodeToTestifInLoopAndBailOut() 

    profiles.clear(Damage)  
    profiles.add(Damage(EM, Amount=profiles.get(DamageTaken(EM))
    profiles.add(Damage(Kin, Amount=profiles.get(DamageTaken(Kin))
    profiles.add(Damage(Therm, Amount=profiles.get(DamageTaken(Therm))
    profiles.add(Damage(Exp, Amount=profiles.get(DamageTaken(Exp))

    # Runs the rules to move the RAH resists around and spits out Adjusted values
    CodeToCalculateRAHAdjustment()

    profiles.clear(Adaptive)
    profiles.add(Adaptive(EM, Amount=AdjustedEM)
    profiles.add(Adaptive(Kin, Amount=AdjustedKin)
    profiles.add(Adaptive(Therm, Amount=AdjustedTherm)
    profiles.add(Adaptive(Exp, Amount=AdjustedExp)

I think that doing it inside Eos itself starts to make things complex. We have issues with this in current Pyfa (where old Eos won't calculate stacking penalties correctly, for example).

By moving simulation and time based logic outside of Eos itself, we can calculate all sorts of things over time quite easily, and Eos will run very quickly as it only needs to calculate that one snapshot. We can also avoid calculating things that are very resource intensive inside Eos (which simulations are) unless they are specifically requested.

Pros:

• Eos stays extremely simple, only deals with a single set of calcs and no complex loops/logic
• Can calculate specific calculations for specific simulations as requested without having to calculate all of them

Cons:

• Requires a framework/skeleton to handle simulations

Projection (Ships, fits, and modules)

Posted by @Ebag333

Currently doesn't seem to be any way to project something onto a fit.

We need to be able to project entire fits. We may also want to project single modules. Pyfa does this currently, but it's often not particularly useful as it just uses base stats which are often far off from the actual stats (especially when bonused hulls are in play).

We could do something like:

fit.project.add(Fit(2446, state=State.hostile))
fit.project.add(Fit(3447, state=State.friendly))
fit.project.add(Module(5443, state=State.Active))

We need to support multiple states.

For modules (if we want to support them, which again I think is of limited use) we just need a state of:

• Online
• Offline

For fits we need four states:

• Offline
• All
• Friendly
• Hostile

The reason for the extra states is that it is currently painful to project fits when they have mixed modules in Pyfa. If my command ship alt has links and neuts, and I project it onto my Vengeance, then my Vengeance will show that I am getting neuted when that will never be the case.

This will become especially bad with the way that CCP is planning on command bursts, where you might have 10+ ships applying command bursts. By being able to sets a projecting ship to friendly, you only allow the beneficial modules to apply (bursts, logi, remote sebo, etc).

Conversely, if you set it to hostile, you can filter out any friendly modules.

Alternatively, we could separate out online/offline from friendly/hostile/all. In current Pyfa we already had the online/offline code, so @blitzmann and I had discussed simply adding a new column for friendly/hostile/all so we didn't have to touch online/offline.

Not sure which is easier in Eos, but either solution would work.