[Proposal] Move device projects to respective vendor folder

[dotnet]

[Proposal] Move device projects to respective vendor folder

[joperezr]

another similar option that @richlander and I talked about in the past, was to have subfolders for device binding types, like for example: weather sensors, motion sensors, analog-to-digital converters, etc.

[joperezr]

Also I plan on submitting a PR with a contribution doc outlining all of this later today 😄

[shaggygi]

@joperezr Since we're still in the early stages of bindings, let's go ahead with what @joshfree is promoting and circle back on results. I guess I can't vision this right now. Maybe it will click with me after a few binding libraries are made and markdown created.

[shaggygi]

@krwq I think the list should work for now. Your script seems to do a good job on listing/displaying data. I'll close for now, but could see the topic arising again as many bindings become available. Thx

[joperezr]

That is a very interesting suggestion, What do you think @richlander?

As you suggest, it would be good to have this for the contribution doc, since the whole purpose of that is to show the community how to contribute in three different areas: Adding support for more boards (eg: beaglebone), adding more libraries that control specific sensors or devices, and contributing with our main library.

[shaggygi]

Also, this topic might help structure the components' tests and samples folders. While I think it is a good practice to have a top-level samples folder with the basics (e.g. toggle LEDs, etc.) where newbies can quickly start using/contributing, it seems like the individual device binding projects would have their own consistent convention for tests/samples, as well.

I honestly don't know the best solution for this as it could get complicated quick. The reasoning is the different OSes, maker boards, drivers, etc. we need to test. It seems like the top-level testing of main APIs (GPIO, I2C, SPI, etc.) would cover the majority of functionality expected (e.g. read, write, etc.). However, the components will have specific terminology like configurations, commands, registers, etc. where we want to make sure it is functioning correctly.

So the question is how would the binding projects' structure be?

1. Should there be a simple unit test project to cover the components basics? I'm assuming we could mock the OS and board in this test project. Guidance would be nice on doing this.

Mcp3008 (top-level project folder)
- Mcp3008 (folder with Mcp3008.csproj)
- Mcp3008.Tests (folder with Mcp3008.Tests.csproj that includes the basic unit tests)

2. How would we structure samples? Would we structure by OS > board or board > OS? It's a 🐔 or the 🥚 scenario. Or do we just keep all in a single respective samples folder. This might be getting a little too granular. I'm thinking most devs will want to see a quick reference (no matter what OS/board) and then start hacking away with the API with what hardware they have.

Mcp3008 (top-level project folder)
- Mcp3008 (folder with Mcp3008.csproj)
- Mcp3008.Tests (folder with Mcp3008.Tests.csproj that includes the basic unit tests)
- samples (folder with all related sample projects for Mcp3008)
-- ReadSingleEndedChannel.csproj (Reads a single-ended channel using Win10 on a RPi3)

Again, I don't know the best option here and opened to anything. I just think it would be good to determine a standard before being too late in the game. Thoughts?

Thx

[joperezr]

it seems like the individual device binding projects would have their own consistent convention for tests/samples

For this I was thinking that every new device should have an .md file (sort of like this one) where people could find snippets with sample code for the general usage. The main reasoning for this is that we won't be building those samples (as how you already mentioned, this could get complicated quickly and we don't want to fail the build for a failed device sample build) so I guess it is better to instead have this as an md file.

I also think that kind of the same logic applies for the device bindings tests. For these, we are not planning (at least for now) to run specific unit test projects that could fail the build, so the only tests we plan to run (at least initially) are the ones for the main library (System.Device.Gpio).

What do you think @richlander ?

[shaggygi]

@joperezr Sounds reasonable. You made a good point on failed builds I wasn't thinking about. READMEs seem like they would be enough to get makers started. I guess I was thinking it would be helpful to get others started quickly without them creating a new project and other initial hurdles on their own. Maybe once this project becomes a little more fleshed out, I could see having a mini-series on Channel9 that provides key points. This was a good one that helped me.

I still like the idea of at least having a respective test project for the binding (as mentioned above with example of Mcp3008). Maybe not for actual integration testing with hardware, but mainly for bindings internal logic. I'm trying to get familiar with developing another binding (future PR) and this component has different controlling bit patterns that warrant tests. I'd be happy if there is guidance on consistent mocking of the protocols within this level of testing.

Is there a way to ignore the bindings and their test projects so they don't mess with the main builds?

Thx

[joperezr]

Actually now that I've given it more thought, if we have the sample code only on the .md file then it could go stale pretty quick since people could then change the API but never come and change the samples. For that reason, I think it would be a good rule of thumb to enforce a sample project with quick consumption examples for every new binding. For tests, I wouldn't enforce them because if you take a look at the example of the MCP3008, the only public facing api it has is Read() which requires an actual device if you want to run tests for it, but as you point out, there are other bindings that will require some unit tests that won't require hardware.

Basically the summary is: If you are adding a new binding you need to A) also add a samples project beneath it, B) Optionally you could decide to add a unit-test project in case you don't require an actual sensor/device in order to run those tests. Also, initially I will reconsider what I said above and just build samples as well and build and run tests as part of the main build. If we start seeing that this is causing problems (slow build times, sample projects constantly breaking build) then we will reconsider. This is more or less how a new binding layout would look like:

iot/
  src/
    devices/
      MyNewBinding/
        MyNewBinding.csproj
        MyNewBinding.cs
        README.md   <-- This should include a fritz diagram of an example
        samples/
          MyNewBinding.Sample.csproj
          MyNewBinding.Sample.cs
        tests/   (<--  Optional)
          MyNewBinding.Tests.csproj
          MyNewBinding.Tests.cs

What do you think @shaggygi?

[shaggygi]

@joperezr sounds great. I think it would still be beneficial to gather more input/discuss original issue topic related to vendors folder.

Until then, I will try to follow your structure above on a binding example. This will be a good exercise as I've not used the fritz tooling.

Thx a bunch!

[joshfree]

While we can organize by vendor; I would vote to start out initially by structuring the bindings into directories by capabilities (and eventually have the data categorized both by vendor and by capability in the root readme -- this sounds like a nice problem to have 😄).

What we start with initially will invariably be restructured a number of times as the quantity of device bindings grows over time.

Here's my quick and dirty proposal on directory structure (assuming the inner directories then contain what @joperezr wrote above):

iot/
    src/
        devices/
            audio/
                equalizer/
                midi/
                mp3/
                synthesizer/
            sensors/
                atmospheric/
                biometric/
                distance/
                imaging/
                movement/
                rfid/
            motors/
                servo/
                solenoid/
                stepper/
            displays/
                lcd/
                oled/
                touch/
....

[shaggygi]

@joshfree so would the Mcp3008 binding structure be moved to something like the following?

iot/
    src/
        devices/
            sensors/
               adc/
                  Mcp3008/

This sounds reasonable and a good place to start where we could always restructure later. My only concern on possibly restructuring later is links/references that break in issues/markdown/wikis/blogs.. I'm assuming bigger teams (e.g. CoreFx) has had this problem before, though. This might not be a big problem to overcome.

I'm favoring vendors because there are cases where hardware is offered that include a mixture of features. Here are some examples:

1. Features - There could be a home automation module that includes touch display, temp/humidity sensors, a few I/O and magnetic swipe reader. How would we categorize this example? We might be able to use a tagging strategy to help users search/find a specific type of feature (e.g. touch, gpio, servo, adc, etc.) while still having the hardware under the respective vendor. Bindings could have multiple tags in a markdown (or elsewhere) to represent all features for a device. I'm not sure on a good tagging method in markdown, though.

In addition, chips will many times provide multiple capabilities. For example, there might be a Serial-to-I2C converter chip, but also includes a few digital and analog I/O. Would this be labeled under...

iot/
    src/
        devices/
            sensors/
               serial/

or...

iot/
    src/
        devices/
            sensors/
               gpio/

2. Multiple Components - There are scenarios where makers will have a mixture of components on a board. For example, it could include I2C EEPROM, digital logic decoders/demultiplexer with GPIO, GPS module (that uses a Ublox component), and SPI accelerometer. I'm not sure the best location for this hardware based on the variety of features.

In most cases, the maker will offer a higher-level API utilizing the core System.Device under the covers. However, there might be cases were we could "kill 2 birds with 1 stone". For example, there could a) already be bindings the maker uses in their bindings or b) the maker develops & and releases their's to community with new additional bindings developed during the process that enable users to benefit for other projects (buiding the eco-system 😉 😉).

For giggles, let's say ShaggyInc offered hardware (GizmoIoT) with its own binding. However, behind the scenes, it referenced the bindings for the 4 different vendor components. Using the vendors structure, it helps to separate out between individual component and a maker that may or may not have used other provided bindings.

Also note (and I'm not completely sure as I just randomly picked hardware), but in my example below points out that the SparkFun GpsRtk module uses the Ublox NeoM8P component. I'm assuming the interface to control the module is dumb-downed to control on the SparkFun GpsRtk. Therefore, it's binding would possibly be simpler to use and the Ublox NeoM8P binding (if one developed) would be a little more complicated if used individually like other components (e.g. Microchip 24xx512).

iot/
    src/
        devices/
            vendors/
                AnalogDevices/
                    ADXL345/        // I2C accelerometer
                Microchip/
                    24xx512/        // SPI EEPROM
                ShaggyInc/
                    GizmoIoT/       // A made up module (offers own binding that uses ADXL345, 24xx512, GpsRtk and SNx4HC138 behind the scenes)
                SparkFun/
                    GpsRtk/         // GPS module  (has Ublox NeoM8p component, but may not be using the NeoM8p binding below)
                TexasInstruments/   
                    SNx4HC138/      // Digital logic
                Ublox/
                    NeoM8P/         // GPS controller
    

Does this make sense?

Thx for reading this far 😄

[grahamehorner]

While all the above makes sence IMHO I believe that the System.Device.xxx shouldn't have anything other than basic implementations and other vendors should support a given type of communication protocol should implementation their driver/controller in a similar manor exposing a common interface that can be built upon. If a vendor/hardware device supports multiple communication protocols then they would implement multiple interfaces by layering accordingly, the results could be a type of nuget packages with special tags that indicate hardware and platforms supported by the package.

[joperezr]

@shaggygi I see where it could be confusing for devices that have more than one feature. What about having a markdown dictionary as @joshfree suggests that has both categorizations (by vendor and by capabilities) where the same device can be in more than one capability category? Then, if we go with the model of sorting them by capabilities it should be ok as long as you pick a folder of one of the n capabilities supported by the sensor. The reason why I would still prefer this rather than having it by vendors is that I think there will be more people looking for sensors by capabilities than by vendors.

@grahamehorner I believe what you said above is exactly the plan. This repo contains two different types of libraries: The System.Device.Gpio library (which is the main library that has support for I2C, SPI, GPIO and PWM), and the device binding libraries that will be implemented on top of System.Device.Gpio library. Initially, we only plan to have a NuGet package for System.Device.Gpio library, while we will do source distribution for device binding libraries, but in the future, we might have separate NuGet packages for these as well.

[shaggygi]

Related to sample project names, it seems like there won't always be just one [binding].Sample.csproj. (hence the folder name samples). Maybe this could be the default and bare basic example (would probably be the only one in most cases). However, I can see multiple sample projects for bindings, especially for more complex devices. Even if we stick with having all different examples (e.g. ReadWithSpi, ReadWithGpio, etc.) within the default sample project and commenting out method calls for flow control, this would allow future growth of binding sample projects. Each project sample folder would allow for specific assets (e.g. READMEs, Fritzing files, pngs, etc.) to be separated/managed, as well.

For example...

iot/
  src/
    devices/
      MyNewBinding/
        MyNewBinding.csproj
        MyNewBinding.cs
        README.md   <-- This should include a fritz diagram of an example
        samples/
            MyNewBinding.Sample    // Folder for default and general use of device.
                MyNewBinding.Sample.csproj
            MyNewBinding.TriggerVoltageBrownoutInterrupt   // Folder for another sample
                MyNewBinding.TriggerVoltageBrownoutInterrupt.csproj
            MyNewBinding.NotifyAzureEventGrid    // Folder for another sample
                MyNewBinding.NotifyAzureEventGrid.csproj
            MyNewBinding.DoSomethingWithDocker    // Folder for another sample
                MyNewBinding.DoSomethingWithDocker.csproj

In this case, we would need to move Mcp3008.Sample.csproj sample down into its own folder.
Thoughts?

[joperezr]

Personally, I think it's better to only have the one project, which in turn has several different source files (One source file for each sample type). And we could also have multiple samples in one same source file in case they are related. The only reason is so that we don't get that many .csprojs since in theory that will slow the build if we have to call msbuild and the compiler more times than if we just call it one per binding (even if you pass in more than one source file). Restore will also take longer.

[shaggygi]

Here is some more categories based on @joshfree thinking. Not complete.
Used TI and Microchip as a reference.

I still believe the bindings' project namespaces should following the path as there will be many added overtime and should have some separation.

 Iot
    Device
        Amplifier
            Comparator
            CurrentSense
            Instrumentation
            Operational
            ProgrammableGain
            PowerSense
            SelectableGain
            SpecialFunction
            VariableGain
        Audio
            
        ClockTiming
            ClockGeneration
            ClockDataDistribution
            Oscillator
            RealTimeClock
        DataConverter
            AnalogToDigital
            DigitalPotentiometer
            DigitalToAnalogConverter
            EnergyMeasurement
            FrequencyToVoltage
            VoltageToFrequency
            VoltageReference
        Interface
            Can
            Infrared
            IoExpander
            Lin
            Usb
        Isolation
        
        Lcd
            
        Logic
            BufferDriver
            FlipFlop
            Gate
            Latch
            Register
            Transceiver
            Specialty
            VoltageLevelTranslation
        Memory
            OtpEeprom
            ParallelEeprom
            SerialEeprom
            SerialEeram
            SerialFlash
            SerialSram
        MotorDriver
            Actuator
            BrushedDc
            BrushlessDc
            OpticalDisk
            Solenoid
            Stepper

        PowerManagement
        
        RfMicrowave
        
        Sensor
            AmbientLight
            Humidity
            Magnetic
            MillimeterWave
            SignalConditioner
            Temperature
            TimeOfFlight
            Ultrasonic
        
        SwitchMultiplexer
        
        Wireless

[krwq]

I haven't read the whole thread but please note that some devices have more than 1 types of sensors included (i.e. temperature and pressure).

Other example is SenseHat which bundles up sensors and devices together.

What I'd vote to do instead is perhaps add some text file in each directory which would contain 1 line per tag and each tag would be i.e. thermometer, temperature, DS18B20, 1-wire. We could perhaps create some UI for that so that it is easier to pick tags

That would allow easy searching and allow us to generate some docs/index based on that. However I can think of directories there will be always something which doesn't fit there (vendors could work but some sensors can be very similar but branded/sold by different vendors).

[shaggygi]

Yeah, this is a tricky one, but definitely one that we need to think through to be a good maintainable story. I'm still scratching my head on how to possibly structure 😖 😕

[krwq]

tags ftw! 😄

[krwq]

@shaggygi now that we have device listing how do you feel about this issue? Do you think we should still change this structure? I still hasn't come up with anything better than we have already so I'm a little bit leaning toward closing this since we don't have any clear plan forward.

[krwq]

agreed, if this becomes a problem again then let's reopen or recreate this issue