spiwn/factorioapiscraper

A small script that goes through the Factorio API documentation html and exports a json intented to be used by autocomplete extensions such as https://github.com/simonvizzini/vscode-factorio-lua-api-autocomplete

factorioapiscraper's Issues

context.json data model

Hi I've been working further to ingest context.json and produce my own flavor of outputs. In that process I have noticed a couple of inconsistencies/issues/potential improvements to the data.

I'm able to work around/special case these, but I'm reporting them to you for your information.

LuaItemStack https://lua-api.factorio.com/latest/LuaItemStack.html has a couple of return types that might require some more parsing or special handling. Specifically it produces types with spaces in them.

"type": {
  "value": [
    "blueprint entity"
  ],
  "type": "Array"
},

CustomDictionary and CustomArray are encoded like

"type": {
  "value": [
    "CustomDictionary",
    "string",
    "LuaTechnology"
  ]
},

and

"type": {
  "value": [
    "CustomArray",
    "of",
    "Alignment"
  ]
},

while they might be more consistenly represented like

"type": {
  "type": "CustomDictionary",
  "value": [
    "string",
    "LuaTechnology"
  ]
},

to match

"type": {
  "type": "Table"
  "value": [
    "string",
    "uint"
  ],
},

Ideas and suggestions mostly about defines, events and json

By accident i found out that if you do

---@type table<string, number>
---@class defines.foo
defines.foo = {
  bar = 0,
  baz = 0,
}

You don't only get intelisense for the fields and the class, but you can also use for k, v in pairs(defines.foo) do end and it knows the types of the keys and values. It might be worth doing since sometimes you do loop through defines.

On that note, note how i used a table constructor. You can save quite a bit of file size using those instead of indexing all the way for every field. It would also make sense for that to load faster, but i have not measured it. I've already tried it and it's not too hard. It just means that the generic writeSingleDefine function doesn't make that much sense anymore, but i've not looked into how neccessary it is to be generic and recursive.

You can also see that i defined a class called defines.foo. This can sometimes also be wanted/needed, in fact the api docs themselves use it sometimes to refer to all values an enum may have.

Also on the topic of defines i don't believe it makes sense to put the event parameter classes in there and name them exactly like the define would be named, though the name is really whatever. Either way i'd like the defines to be purely enums (not talking about defines.prototypes for now) and have the event parameters be in a separete file/location. It just doesn't make logically sense for them to be there. This is a bigger deal for the json format (=> data structure) because when i tried to use that it got more complex than it needed to be trying to get the actual event parameter definitions. I hope this makes sense.

About the json: i'd also move the event stuff out of defines.json, which would come naturally with the previous change. And in general my idea behind the json format would be that it contains as much raw data as possible where using it doesn't require any additional string parsing or data clean up. I believe that is what you have done already, but i just wanted to have it said.

That is all i've run into or noticed so far. Important to note is that these are all my opinions and or wishes and i'm open for discussion about any of them.

Generate LuaDoc

Generate LuaDoc from the Factorio API documentation

sumneko/lua-language-server has support for annotated types. This scraper can generate appropriate .lua files so the Visual Studio Code extension can provide auto-completion, documentation and anything else the language server supports.

Generated errors, warnings and infos

Errors

Some function parameter names are invalid identifiers (such as end, function and containing .). In parseParameter() those need to be converted into valid identifiers, most likely by replacing all invalid characters with _ and appending an _ to keywords. Valid identifiers match the regex ^[a-zA-Z_][a-zA-Z0-9_]*$, so basically replace everything that isn't [a-zA-Z0-9_] and prepend with an _ if the first char is a number.

Warnings

All defines are getting the undefined global number assigned to them. Even if that is currently a placeholder, it either needs to be a defined global or local or (imo preferably) just 0.

---

All the basic types generate undefined-doc-name warnings. Aliases seem to just replace the name you actually want, but we either have to take that compromise or we try out this and see how it goes:

---@class float : number
---@class double : number
---@class int : number
---@class int8 : number
---@class uint : number
---@class uint8 : number
---@class uint16 : number
---@class uint64 : number

(deriving from integer doesn't work because that's apparently just an alias for number but i know it's at least a bit more special than that. just how the class number is more special than just deriving from the class any which doesn't have anything defined. Just language server internals looking weird i guess.)

Infos

If no-implicit-any is enabled:

"Lua.diagnostics.neededFileStatus": {
  "no-implicit-any": "Any",
},

Lots of those warnings get generated. it most likely makes the most sense to disable this completely for all files by adding ---@diagnostic disable: no-implicit-any at the top, though it might be useful for finding issues during development.

The Rest

Once those are all fixed you should be left with 9 errors and 28 warnings (with just lowercase-global warnings disabled in Lua.diagnostics.disable). All of those are luadoc-miss-type-name and undefined-doc-name and looking through them they are all things that also need special treatment.

(How do i know? I hack fixed all of these issues. Initially to see if performance (being diagnostic speed) would be better with all of them fixed and it was noticably better, but still not great.)

Edit 1: you'd probably have a few more warnings than 28, because i have the defines.foo classes defined.
Edit 2: oh you'd have way more than that. about 100 or so. In order to reduce warnings for performance testing i did add

---@class CustomDictionary
---@class CustomArray

Planned: Automatic builds/runs

Automatic scheduled runs to check for newer Factorio api versions and create documentation for them.
Using Github Actions as the running environment.
Maybe using logiqx/python-bs4 docker image as it covers the scraper requirements (python, bs).

I am planning to do it within 2-3 weeks.