Expand to cover all endpoints and devices

[fhscholl]

Full inventory of SwitchOS and SwitchOS Lite endpoints. Added endpoint alternates for renamed endpoints in SwitchOS 2.18. Added full synthetic testing data based on JS extracts from all SwitchOS and SwitchOS Lite device firmware.

Supported Firmware

SwOS

Device	Versions
CSS106	2.16, 2.17, 2.18
CSS305	2.16, 2.17, 2.18
CSS305r2	2.16, 2.17, 2.18
CSS309	2.16, 2.17, 2.18
CSS310	2.16, 2.17, 2.18
CSS310G	2.16, 2.17, 2.18
CSS312	2.16, 2.17, 2.18
CSS317	2.16, 2.17, 2.18
CSS318FI	2.16, 2.17, 2.18
CSS318G	2.18
CSS318P	2.16, 2.17, 2.18
CSS320P	2.17, 2.18
CSS326	2.16, 2.17, 2.18
CSS326Q	2.16, 2.17, 2.18
CSS326XG	2.16, 2.17, 2.18
CSS328	2.16, 2.17, 2.18
CSS328P	2.16, 2.17, 2.18
CSS354	2.16, 2.17, 2.18

SwOS Lite

Device	Versions
CSS606	2.21
CSS610	2.18, 2.19, 2.20, 2.21
CSS610G	2.18, 2.19, 2.20, 2.21
CSS610OUT	2.21
CSS610PI	2.18, 2.19, 2.20, 2.21
FTC11XG	2.18, 2.19, 2.20, 2.21
FTC21	2.19, 2.20, 2.21
GPEN21	2.18, 2.19, 2.20, 2.21
GPER14I	2.19, 2.20, 2.21

• Add local copy of analysis data

[fhscholl]

Sorry for the giant PR. I found that if you download firmware from MikroTik, you can extract specific components, including the web frontend and engine.js. Using that, I created a full knowledge base and analysis of the different devices for the current and previous few firmware versions.

I included everything for completeness, but some should be excluded from the Home Assistant integration and used only for diagnostics.

[probert94]

Thank you again for your contribution. I'll take a look at it as soon as possible but it's quite a lot to review.
Would you mind sharing some information on how you extracted all this information from the firmware?
Do you think it makes sense to extract the knowledge base into it's own repository so that it could be used by different libraries independent of the used language?

[fhscholl]

The results of the analysis of each firmware are contained in tests/tools/all_analyses.json

The firmware files are a collection of binary files stuck together. Some firmware chunks are encrypted or unreadable, but they don't matter because they are for device execution. Also in the files are gzip-compressed files that are copied and decompressed as the firmware is flashed to the device. I have a simple routine that scans the bin file, searches for gzip magic numbers, and tries to extract files from those locations.

On SwOS Lite, the extraction gives two files: index.html and engine.js.
On SwOS, the extraction produces a single file, web-interface.html, which contains the engine.js logic as an embedded script.

I extracted them all and ran them through a similar discovery process to that used for the previous changes. Most of them are very similar with minor differences for port counts and device capabilities.

---

The vast majority of changes in the PR are test files for each device and firmware version. If you want me to exclude those or reduce the file count, I can do that.

[probert94]

Thank you for the explaination, thats a pretty smart approach!
I had a quick first look at them and noticed, that the property names for SwitchOS are missing. So right now the new properties and endpoints can only be used for SwitchOS Lite.
The all_analyses.json is quite interesting, I have a few questions and suggestions here:

• For SwitchOS only the field id is listed, all other fields are missing.
• Models with dual boot (can be used with RouterOS or SwitchOS) are missing (example: CRS326-24G-2S+).
  List of devices on their website
• The fields and endpoints are currently unrelated. I guess it would be great to have a structure like:

  "enpoints": {
    "link.b": {
      "name": "Link",
      "fields": {
        "i01": {
          "name": "Enabled",
          "type": "F"
        }
      }
    }
  }

• The field types could be mapped to a meaningful name, for example "F" -> "bool_list"
• The features don't seem to match the available endpoints. At least for the fist model (ftc11xg), the features list "poe", while the endpoint is missing.

Regarding the tests: Tests are an important part of every software and I really appreciate your work here. Unfortunately I am very new to python (this library is my first real contact with the language) and I didn't touch testing yet. That said, it's pretty hard for me to review those changes, not only cause of the huge amount of files but also cause of the lack of knowledge on my site. So I think for this PR it would be better to keep tests limited to 1 SwitchOS and 1 SwitchOS Lite model for now and add additional devices in a later PR.

Thank you again for your awesome work!

PS: If you prefer to discuss this outside of Github, feel free to send me a mail!