armmbed/ci-test-shield

mbed CI Test Shield

ci-test-shield

This repo contains the test code for the ARM mbed CI Test shield. For the hardware schematics please see the mbed HDK. Make sure to use the latest version of the hardware to be fully compatible with the tests!

If any tests fail then your platform is exhibiting unexpected behavior. More often than not this indicates a bug in your software port. For maximum mbed compatibility all tests should pass on your platform.

If you platform does not support a specific feature, such as AnalogOut, then that test will fail or error out. This is expected behavior, please make sure to note this in your communications.

What to use the CI Test Shield for?

The CI Test Shield has several uses.

1. Test mbed-os software port. Useful for mbed partners and community contributors.
2. Continuous Integration testing during development and during maintenance release cycles.
3. SD Card storage can be used for firmware update development.

What is tested?

Current API's that are tested by the CI Test shield are as follows

1. DigitalIO, InterruptIn, PwmOut - tested by Loopback on pins D0-9
2. AnalogIO - tested by resistor star network.
3. I2C - tested with temperature sensor and EEPROM memory.
4. SPI - tested with SD Card.
5. UART - tested by greentea working

Coming Soon

• Ticker/Timer/Time

Hardware : Where to get CI Test Shield?

1. Buy pre-made shield - MinewTech , LTEK (Coming Soon)
2. DIY - buy PCB from OSHPark and parts from Octopart and assemble it yourself!
3. Custom - use the mbed HDK to modify the design to match the headers on your board.

Software : How to use?

To run the tests associated with the ci-test-shield follow these steps:

Pre-requisites

0. Install mbed-cli tool (need version 1.0.0 or greater). Please verify that you installed mbed-cli requirements
1. Install mbed-os requirements
2. mbed import https://github.com/ARMmbed/ci-test-shield.git Clone this repo to your computer using mbed import.

Usage

0. cd ci-test-shield- Navigate to the repo on your command line.
1. mbed target auto - Automatically detect board plugged into computer and set it as the target
2. mbed toolchain GCC_ARM - Select compiler. You can also use IAR, UVISION, or any other toolchain listed in mbed toolchain --supported
3. [Optional] Update mbed-os and tests to latest version. Do nothing to use the last stable version.
4. mbed test -n tests-* --app-config .\mbed_app.json - Run CI Test Shield tests.

Customization

All pin mappings are done in the mbed_app.json file. If you need to remap or override you can do so here. Please note that all pin mappings will be pre-pended with MBED_APP_CONF_.

Different Pins

If your board has peripherals on pins that differ from the Arduino R3 Header layout you will need to manually fly wire them into the appropriate header and adjust the mbed_app.json file.

For example: the NXP K64F board does not have AnalogOut on A5, instead it is on an inside header pin called DAC0_OUT that is not broken out to the Arduino R3 Headers. To account for this we manually fly wire the DAC0_OUT pin to the A5 pin on the CI Test shield and add the following to the target_overrides section of the mbed_app.json file.

    "target_overrides": {
        "K64F": {
             "AOUT": "DAC0_OUT"
        }
    }

The above code essentially says "if the target is K64F then #define AOUT DAC0_OUT", this allows you to fly wire pins and only modify things in one file instead of having to modify all the tests themselves.

Non-Arduino Header

If you are testing a board that does not have Arduino R3 style headers then you will need to either fly wire all the pins across manually, or if you want a more permanent solution spin your own version of the CI Test shield with headers that map to your board.

Either way, you will need to heavily modify the mbed_app.json file and redefine every pin mapping since your platform will not have the Arduino Header D0-15 or A0-5 pin aliases.

For more on how config files work see the mbed OS docs.

Troubleshooting

• Use the -v flag for verbose debug and failure messages that can help you troubleshoot.
• To enable debug messages, set DEBUG_MSG to 1 in the ./mbed_app.json file
• Try running a clean build with mbed test --clean -n tests-* --app-config .\mbed_app.json -v
• Make sure your board is detected to run tests. Use the mbed detect command to verify your board is recognized and has a COM port assigned. If the Serial driver isn't working then tests cant run.
• make sure you are using the latest version of greentea and mbed-cli. Try running pip install -U mbed-cli mbed-ls mbed-greentea to update them all to their latest versions.
• Tests in the assumptions folder should all pass, if any of the assumptions tests fail then their associated test will likely also fail. For example, AnalogIn pins should also be DigitalOut capable for the AnalogIn tests to work, so the assumptions test for AnalogIn tests this functionality.
• If all tests are failing try bending aside pins D0/D1 on the CI Test Shield. Some platforms only have 1 Serial peripheral and will tie together the debug serial channel and the loopback serial channel, which causes problems for the communications for the testing platform.
• Make sure to put an SD Card into the microSD slot for SPI Tests to work.
• If you are seeing any weird problems please contact [email protected] for help.

Known Issues

• The latest version of DAP-Link interface firmware should be used as old versions may not support the testing framework.
• AnalogIn pins must also support DigitalOut
• Tests require mbed-cli version > 0.9.9 .

Repository structure

The latest version of the hardware and software is available on master. Development should not occur on this branch - all development should be done on the corresponding hardware branches. Only commits tagged for production will be moved over to master.

Each version of the hardware has its' own branch.

• HW0 - Board identity: 0.1.0. Original hardware board.
• HW1 - Board identity: 1.0.0. Revamped version of HW0 which has PCB resistors, new EEPROM chip.
• HW2 - Board identity: 2,0,0. Latest and greatest, removed UART resistors. Updated FLASH.

Developement should take place on the latest hardware branch. Currently that hardware branch is HW2. When a feature is approved and tested, the hardware branch will get tagged and merged into master.

Release numbers are also now in effect. Tags are using the semantic versioning model. X.Y.Z

• X: Hardware version. Software for each hardware version are incompatible with previous hardware versions
• Y: Feature. New features increment the minor number
• Z: Patch

More information

For more information see the Releases page.

License

Apache 2.0 - feel free to re-use, tweak, modify for personal or business use.

ci-test-shield's Issues

Linking error due to Block Drivers now being included in Mbed

Linking error log here: https://pastebin.com/uECQ2Kaw

Debug logs usable within interrupt routines

#31 removed printf rom one ISR handler. This debug msg is useful to have, just can't use printf as it is dangerous for various reasons. We should implement here a replacement that is interrupt safe.

Production/Alternate test header

@maclobdell

We have decided to use a PCIE-x1 edge connector as our standard test/debug header.
We've broken out SWD, multiple serial ports, QSPI, USB, power, etc.

Not having the time nor intimate knowledge of testing for Mbed OS, we have not put a great amount of thought into which signals and where to put them on the header

Considering what seems to be a lack of formal specification for the Arduino header, would it be beneficial to specify an alternative test/debug/expansion header for Mbed OS boards?

As of this moment we will probably be adding an Arduino header to our CI/test/debug board in order to support the current Mbed ci-test-shield over our PCIE-x1 connector.

K64F Fails to read I2C EEPROM

After fixing the other unrelated issue: #16

The I2C EEPROM tests fail for K64F.

+--------------+---------------+---------------+------------------------------------+--------+--------+--------+--------------------+
| target       | platform_name | test suite    | test case                          | passed | failed | result | elapsed_time (sec) |
+--------------+---------------+---------------+------------------------------------+--------+--------+--------+--------------------+
| K64F-GCC_ARM | K64F          | tests-api-i2c | I2C -  EEProm WR 10  Bytes         | 0      | 3      | FAIL   | 0.5                |
| K64F-GCC_ARM | K64F          | tests-api-i2c | I2C -  EEProm WR 100 Bytes         | 0      | 1      | FAIL   | 0.8                |
| K64F-GCC_ARM | K64F          | tests-api-i2c | I2C -  EEProm WR 2 Bytes           | 0      | 3      | FAIL   | 0.47               |
| K64F-GCC_ARM | K64F          | tests-api-i2c | I2C -  EEProm WR Single Byte       | 0      | 2      | FAIL   | 0.31               |
| K64F-GCC_ARM | K64F          | tests-api-i2c | I2C -  Instantiation of I2C Object | 1      | 0      | OK     | 0.06               |
| K64F-GCC_ARM | K64F          | tests-api-i2c | I2C -  LM75B Temperature Read      | 1      | 0      | OK     | 0.12               |
+--------------+---------------+---------------+------------------------------------+--------+--------+--------+--------------------+
mbedgt: test case results: 4 FAIL / 2 OK

After the I2CEeprom library does a write to the EEPROM, it does a sequence of 0-byte writes every 1ms until it gets a ACK indicating the write finished on the EEPROM and it is ready to read. After it gets an Ack, then the read should happen.

But on the K64F the read doesn’t happen.

It seems that, if a 0-byte write cycle results in an Ack, there is no stop condition afterwards, and the next write cycle that is requested results in Bus Busy error.

I think that, for compatibility, the K64F hal implementation/driver needs to ensure that 0-byte write sequences (or all write sequences that are finished?), are followed by stop condition to release the bus.

@mmahadevan108

I2C-EEProm write failed because it requires a little delay

tests-api-i2c got failed.

Log analysis

1. EEProm WR passed when string length is 2 and 10 bytes, but failed when the length is 100 bytes at Case 9.
2. It means that a string, which contains 99 chars and a ending NULL, was written to I2C slave, right after reading 100 bytes data from the same address, but the read string is different from the one written. even read string size is different from written string's.
3. and the failure happened mostly when string size is bigger then page size (32 bytes).

A. The log with more printf, the failed point is marked by <<

[1500793846.72][CONN][RXD] >>> Running case #5: 'I2C - EEProm WR 2 Bytes'...
[1500793846.77][CONN][INF] found KV pair in stream: {{__testcase_start;I2C - EEProm WR 2 Bytes}}, queued...
[1500793846.78][CONN][RXD]
[1500793846.78][CONN][RXD] ****
[1500793846.80][CONN][RXD] Address = 5
[1500793846.81][CONN][RXD] Len = 2
[1500793846.84][CONN][RXD] Num Bytes Written = 2
[1500793846.87][CONN][RXD] Num Bytes Read = 2
[1500793846.90][CONN][RXD] Written String = D
[1500793846.92][CONN][RXD] Read String = D
[1500793846.92][CONN][RXD] ****
[1500793846.98][CONN][INF] found KV pair in stream: {{__testcase_finish;I2C - EEProm WR 2 Bytes;1;0}}, queued...
[1500793847.03][CONN][RXD] >>> 'I2C - EEProm WR 2 Bytes': 1 passed, 0 failed
[1500793847.03][CONN][RXD]
[1500793847.09][CONN][RXD] >>> Running case #6: 'I2C - EEProm 2nd WR 2 Bytes'...
[1500793847.14][CONN][INF] found KV pair in stream: {{__testcase_start;I2C - EEProm 2nd WR 2 Bytes}}, queued...
[1500793847.15][CONN][RXD]
[1500793847.16][CONN][RXD] ****
[1500793847.17][CONN][RXD] Address = 5
[1500793847.18][CONN][RXD] Len = 2
[1500793847.21][CONN][RXD] Num Bytes Written = 2
[1500793847.24][CONN][RXD] Num Bytes Read = 2
[1500793847.27][CONN][RXD] Written String = P
[1500793847.29][CONN][RXD] Read String = P
[1500793847.30][CONN][RXD] ****
[1500793847.35][CONN][INF] found KV pair in stream: {{__testcase_finish;I2C - EEProm 2nd WR 2 Bytes;1;0}}, queued...
[1500793847.41][CONN][RXD] >>> 'I2C - EEProm 2nd WR 2 Bytes': 1 passed, 0 failed
[1500793847.41][CONN][RXD]
[1500793847.47][CONN][RXD] >>> Running case #7: 'I2C - EEProm WR 10 Bytes'...
[1500793847.52][CONN][INF] found KV pair in stream: {{__testcase_start;I2C - EEProm WR 10 Bytes}}, queued...
[1500793847.53][CONN][RXD]
[1500793847.54][CONN][RXD] ****
[1500793847.55][CONN][RXD] Address = 100
[1500793847.57][CONN][RXD] Len = 10
[1500793847.60][CONN][RXD] Num Bytes Written = 10
[1500793847.62][CONN][RXD] Num Bytes Read = 10
[1500793847.66][CONN][RXD] Written String = ABNYCLCBR
[1500793847.69][CONN][RXD] Read String = ABNYCLCBR
[1500793847.70][CONN][RXD] ****
[1500793847.75][CONN][INF] found KV pair in stream: {{__testcase_finish;I2C - EEProm WR 10 Bytes;1;0}}, queued...
[1500793847.80][CONN][RXD] >>> 'I2C - EEProm WR 10 Bytes': 1 passed, 0 failed
[1500793847.81][CONN][RXD]
[1500793847.87][CONN][RXD] >>> Running case #8: 'I2C - EEProm 2nd WR 10 Bytes'...
[1500793847.93][CONN][INF] found KV pair in stream: {{__testcase_start;I2C - EEProm 2nd WR 10 Bytes}}, queued...
[1500793847.94][CONN][RXD]
[1500793847.94][CONN][RXD] ****
[1500793847.96][CONN][RXD] Address = 100
[1500793847.97][CONN][RXD] Len = 10
[1500793848.00][CONN][RXD] Num Bytes Written = 10
[1500793848.03][CONN][RXD] Num Bytes Read = 10
[1500793848.06][CONN][RXD] Written String = EEHBSCLIV
[1500793848.10][CONN][RXD] Read String = EEHBSCLIV
[1500793848.10][CONN][RXD] ****
[1500793848.16][CONN][INF] found KV pair in stream: {{__testcase_finish;I2C - EEProm 2nd WR 10 Bytes;1;0}}, queued...
[1500793848.22][CONN][RXD] >>> 'I2C - EEProm 2nd WR 10 Bytes': 1 passed, 0 failed
[1500793848.22][CONN][RXD]
[1500793848.28][CONN][RXD] >>> Running case #9: 'I2C - EEProm WR 100 Bytes'...
[1500793848.33][CONN][INF] found KV pair in stream: {{__testcase_start;I2C - EEProm WR 100 Bytes}}, queued...
[1500793848.39][CONN][RXD] :75::FAIL: String Written != String Read << failed 1st point! hit TEST_ASSERT_MESSAGE, write string != read string, length: 100 bytes
[1500793848.43][CONN][RXD] :76::FAIL: String Written != String Read
[1500793848.65][CONN][RXD] :77::FAIL: Expected 'YRGCTGXMUZXVMVDVZVCZXGSBXUTRUKGRKQGXPIDXMXDSYMYKAMVCEHGRBWKQVFRTLWAGVRTJVQWBKZJOVPBMHNKFDEZXFWSVUVQ' Was 'YRGCTGXMUZXVMVDVZVCZXGSB@'. String read does not match the string written
[1500793848.65][CONN][RXD]
[1500793848.66][CONN][RXD] ****
[1500793848.67][CONN][RXD] Address = 1000
[1500793848.70][CONN][RXD] Len = 100
[1500793848.73][CONN][RXD] Num Bytes Written = 100
[1500793848.76][CONN][RXD] Num Bytes Read = 100
[1500793848.88][CONN][RXD] Written String = YRGCTGXMUZXVMVDVZVCZXGSBXUTRUKGRKQGXPIDXMXDSYMYKAMVCEHGRBWKQVFRTLWAGVRTJVQWBKZJOVPBMHNKFDEZXFWSVUVQ
[1500793848.92][CONN][RXD] Read String = YRGCTGXMUZXVMVDVZVCZXGSB@
[1500793848.93][CONN][RXD] ****
[1500793849.00][CONN][INF] found KV pair in stream: {{__testcase_finish;I2C - EEProm WR 100 Bytes;0;3}}, queued...
[1500793849.08][CONN][RXD] >>> 'I2C - EEProm WR 100 Bytes': 0 passed, 3 failed with reason 'Test Cases Failed'
[1500793849.08][CONN][RXD]
[1500793849.15][CONN][RXD] >>> Running case #10: 'I2C - EEProm 2nd WR 100 Bytes'...
[1500793849.20][CONN][INF] found KV pair in stream: {{__testcase_start;I2C - EEProm 2nd WR 100 Bytes}}, queued...
[1500793849.22][CONN][RXD]
[1500793849.23][CONN][RXD] ****
[1500793849.25][CONN][RXD] Address = 1000
[1500793849.27][CONN][RXD] Len = 100
[1500793849.30][CONN][RXD] Num Bytes Written = 100
[1500793849.32][CONN][RXD] Num Bytes Read = 100
[1500793849.45][CONN][RXD] Written String = << failed 2nd point! but was not be caught by hitting any TEST_ASSERT_MESSAGE, write string != read string, length: 100 bytes YXTOJMFQQVBGBORSMNLISXWWPSELZKGCPEPDSRDBZZQNYWTBLKZHWAHAOZFOGWVRWAYOONNNAXJBTWDPIDQPQCXIARDIFSQOZPW
[1500793849.58][CONN][RXD] Read String = YXTOJMFQQVBGBORSMNLISXWWPSELZKGCPEPDSRDBZZQNYWTBLKZHWAHAQZNLURZPZOEBVRETQEXUDSNTKVJUVXKQARDIFSQOZPW
[1500793849.59][CONN][RXD] ****
[1500793849.65][CONN][INF] found KV pair in stream: {{__testcase_finish;I2C - EEProm 2nd WR 100 Bytes;1;0}}, queued...
[1500793849.70][CONN][RXD] >>> 'I2C - EEProm 2nd WR 100 Bytes': 1 passed, 0 failed
[1500793849.70][CONN][RXD]
[1500793849.77][CONN][RXD] >>> Test cases: 9 passed, 1 failed with reason 'Test Cases Failed'
[1500793849.79][CONN][RXD] >>> TESTS FAILED!
[1500793849.82][CONN][INF] found KV pair in stream: {{__testcase_summary;9;1}}, queued...
[1500793849.85][CONN][INF] found KV pair in stream: {{max_heap_usage;0}}, queued...
[1500793849.85][HTST][ERR] orphan event in main phase: {{max_heap_usage;0}}, timestamp=1500793849.845000
[1500793849.87][CONN][INF] found KV pair in stream: {{reserved_heap;0}}, queued...
[1500793849.87][HTST][ERR] orphan event in main phase: {{reserved_heap;0}}, timestamp=1500793849.867000
[1500793849.89][CONN][INF] found KV pair in stream: {{end;failure}}, queued...
[1500793849.89][HTST][INF] __notify_complete(False)
[1500793849.90][HTST][INF] __exit_event_queue received
[1500793849.90][HTST][INF] test suite run finished after 4.82 sec...
[1500793849.90][CONN][INF] found KV pair in stream: {{__exit;0}}, queued...
[1500793849.90][CONN][INF] received special even '__host_test_finished' value='True', finishing
[1500793849.91][HTST][INF] CONN exited with code: 0
[1500793849.91][HTST][INF] Some events in queue
[1500793849.92][HTST][INF] stopped consuming events
[1500793849.92][HTST][INF] host test result() call skipped, received: False
[1500793849.92][HTST][WRN] missing __exit event from DUT
[1500793849.92][HTST][INF] calling blocking teardown()
[1500793849.92][HTST][INF] teardown() finished
[1500793849.92][HTST][INF] {{result;failure}}
mbedgt: checking for GCOV data...
mbedgt: mbed-host-test-runner: stopped and returned 'FAIL'
mbedgt: test on hardware with target id: 701100000000000000000000000000000000000097969904
mbedgt: test suite 'tests-api-i2c' ................................................................... FAIL in 62.52 sec
test case: 'I2C - EEProm 2nd WR 10 Bytes' .................................................. OK in 0.23 sec
test case: 'I2C - EEProm 2nd WR 100 Bytes' .................................................. OK in 0.45 sec
test case: 'I2C - EEProm 2nd WR 2 Bytes' .................................................... OK in 0.21 sec
test case: 'I2C - EEProm 2nd WR Single Byte' ................................................ OK in 0.06 sec
test case: 'I2C - EEProm WR 10 Bytes' ...................................................... OK in 0.23 sec
test case: 'I2C - EEProm WR 100 Bytes' ...................................................... FAIL in 0.67 sec
test case: 'I2C - EEProm WR 2 Bytes' ........................................................ OK in 0.21 sec
test case: 'I2C - EEProm WR Single Byte' .................................................... OK in 0.06 sec
test case: 'I2C - Instantiation of I2C Object' .............................................. OK in 0.07 sec
test case: 'I2C - LM75B Temperature Read' ................................................... OK in 0.07 sec
mbedgt: test case summary: 9 passes, 1 failure

Test and Fix

4. I2CEeprom driver writes data with page alignment. please refer following I2CEeprom::write() source code.
5. After testing with different sizes, I identified that

• The failure can be reproduced when the string is written in multiple pages.
• Once the string can be written in a single page, then no error will occur.
• String written in 1st page can be read correctly always.

6. The WR address is 1000, if the string size is 24 byte (=23 chars with a ending NULL), then the string doesn't cross the page ended at 1024. then it can be read correctly. if the string size is 25 byte, then 24 chars will locate at 1st page ended at 1024 and 1 byte (the ending NULL) will be located at 2nd page ended at 1056. then the read and comparison test will fail.
7. Added line A in I2CEeprom::write. a 2ms delay is added before writing in a new page.
   the fix is quite platform dependent. on xxxM066, wait_ms(2); can pass the test, but when changed to wait_ms(1); then the test got failed. I'm not sure the 2ms wait is good enough for other platforms.
8. I assume, other platforms can also have this issue. how were they solved?
9. I2CEeprom driver is not ARM source file, should I make a pull request? or just remark it as a known issue?

B. ci-test-shield\I2CEeprom\I2CEeprom.cpp, with fixed code

size_t I2CEeprom::write(size_t address, const char *buffer, size_t size) {
    // Check the address and size fit onto the chip.
    if (!checkSpace(address, size))
        return 0;
        
    const char *page = buffer;
    size_t left = size;
    
    // Whle we have some more data to write.
    while (left != 0) {
        // Calculate the number of bytes we can write in the current page.
        // If the address is not page aligned then write enough to page 
        // align it.
        size_t toWrite;
        if ((address % m_pageSize) != 0) {
            toWrite = (((address / m_pageSize) + 1) * m_pageSize) - address;
            if (toWrite > size) {
                toWrite = size;
            }
        } else {
            if (left <= m_pageSize) {
                toWrite = left;
            } else {
                toWrite = m_pageSize;
            }
        }
        
        //printf("Writing [%.*s] at %d size %d\n\r", toWrite, page, address, toWrite);
        wait_ms(2);      //line A,  added a little deplay, before writing in a new page.
        // Start the page write with the addres ine one write call.
        char values[] = { (address >> 8), (address & 0xFF) };

        m_i2c.start();
        if (m_i2c.write(m_i2cAddress) == 0) {
            return size - left;
        }
        if (m_i2c.write(values[0]) == 0) {
            return size - left;
        }
        if (m_i2c.write(values[1]) == 0) {
            return size - left;
        }

        // Write the bytes out one at a time to avoid having to copy them to
        // another buffer.        
        for (int count = 0; count != toWrite; ++count) {
            if (m_i2c.write(*page) == 0) {
            // Write failed to return bytes written so far.
                return size - left;
            }
            ++page;
        }
        
        // Stop the transaction now we've completed the page
        // write.
        m_i2c.stop();
        
        waitForWrite();
        
        // Update the counters with the amount we've just written
        left -= toWrite;
        address += toWrite;
    }
    
    return size;
}

Tests - verify that required pins are defined

There can be a case when required pin for instance A1 could be NC (target do not pollute that pin for whatever reason). This would cause failures in tests in later stages. If we test it early, we could identify the problems with pins definitions.

Inspiration from here: ARMmbed/mbed-os#3508 (comment)

mbed HDK link of README is broken

As said in the title ...

Why running the same test on 4 PWMs ?

The PWM API tests run the same test over 4 different PWMs that are expected to be available on
"PWM_0": "D3",
"PWM_1": "D5",
"PWM_2": "D6",
"PWM_3": "D9",
as defined in mbed_app.json

This makes a very strong constraint on the board design and all the boards don't comply with it. Also the tests use PWM_X as the PWM output as defined in mbed_app.json, but the interrupt in input that is used to verify PWM behavior is hard coded in the TESTS/API/Pwm/Pwm.cpp file, as MBED_CONF_APP_DIO_2, 4, 7 and 8 respectively.

Would that be OK to only consider 1 PWM output and respective input rather than 4 and also use a generic entry for the interrupt in input, like "MBED_CONF_APP_PWMIN_0" for instance ?

thanks !

Improve InterruptIn tests

Make InterruptIn tests more verbose. Like this :https://gist.github.com/maclobdell/17dbe75853de7cf8bf9df3e77224bdee

Add UART Tests

Problem

UART Tests cannot be done by Loopback on D0/D1 because some boards (Nucleo / nRF / etc) only have a single UART and use the same peripheral for the Serial Debug lines used by greentea. Thus is these pins are tied together greentea cannot run.

Therefore the UART tests should be redone to use the Serial Debug channel instead of UART Loopbacks.

Solution

Currently not possible

In the future a change should be made to GreenTea that allows for taking control of the serial line and then handing it back. See here : ARMmbed/greentea#196

ci-test for ST platform in latest mbed OS 5.4

I'm now helping a partner to do ci test for their new platform based on STM32F407. In previous mbed versions, they can pass most of the cases but in the latest version, the ci-test just blocked. But other examples such as blinky can work.

I find that in the new version of mbed OS, the ST driver folder is modified. I can repeat this issue on NUCELO_F401RE, I guess the new ST driver structure conflicts with ci-test.

import ci-test repo, use cmd:
mbed target auto
mbed toolchain GCC_ARM
mbed test -n tests-* --app-config .\mbed_app.json

After compile, the log stoped at:

mbedgt: greentea test automation tool ver. 1.2.2
mbedgt: test specification file 'C:\my\dev\mbed_5_0\oob_540\ci-test-new\BUILD\tests\NUCLEO_F401RE\GCC_ARM\test_spec.json' (specified with --test-spec option)
mbedgt: using 'C:\my\dev\mbed_5_0\oob_540\ci-test-new\BUILD\tests\NUCLEO_F401RE\GCC_ARM\test_spec.json' from current directory!
mbedgt: detecting connected mbed-enabled devices...
mbedgt: detected 1 device
mbedgt: processing target 'NUCLEO_F401RE' toolchain 'GCC_ARM' compatible platforms... (note: switch set to --parallel 1)
mbedgt: test case filter (specified with -n option)
        test filtered in 'tests-api-analogin'
        test filtered in 'tests-api-businout'
        test filtered in 'tests-api-digitalio'
        test filtered in 'tests-api-i2c'
        test filtered in 'tests-api-interruptin'
        test filtered in 'tests-api-pwm'
        test filtered in 'tests-api-spi'
        test filtered in 'tests-assumptions-analogin'
        test filtered in 'tests-assumptions-digitalio'
        test filtered in 'tests-assumptions-i2c'
        test filtered in 'tests-assumptions-pwm'
        test filtered in 'tests-assumptions-pwmout'
        test filtered in 'tests-assumptions-spi'
mbedgt: running 13 tests for platform 'NUCLEO_F401RE' and toolchain 'GCC_ARM'
mbedgt: mbed-host-test-runner: started

Add SPI Flash chip

Suggestion

Add SPI Flash chip to CI Test shiled

Reason

Most deployments will use SPI Flash, not SD Cards. It makes sense to add a SPI Flash chip to the next version of the CI Test shield.

Implementation

Add 16MB SPI Flash chip to board, add CS Line to... not D0/D1.

Request to improve the PWM tests in order to support single edge trigger MCUs.

<< Request Information >>

• Contents

Request to improve the PWM tests in order to support single edge trigger MCUs.

• Reasons

• Currently, single edge trigger MCUs cannot test PWM via CI test shield.
  iin.rise(duty_cbfn_rise);
iin.fall(duty_cbfn_fall);

• Not all the MCUs are dual edge triggerable. The single edge trigger MCUs only support either rise() or fall().

• Manual testing is theoretically possible. However, if it is not automated, you cannot check regression for future git submits.

• Manual testing is practically impossible. In order to test, you have to check the precision with an oscilloscope or a logic analyzer, and you have to verify many samples to check the stability.

PWM duty error rate > 5% in tests-api-pwm

SHA
6ca1b89

Description
I meet this issue on NuMaker-PFM-NUC472 and NuMaker-PFM-M453 targets. About the function tests-api-pwm > PWM_Duty_slave, I have some comments:

1. Start PWM waveform recording after PWM configuration has finished. This could avoid recording premature PWM waveform.

   pwm.period((float)period_in_ms / 1000); // set PWM period
   pwm.write(duty_cycle_percent); // set duty cycle
   iin.rise(duty_cbfn_rise);
   iin.fall(duty_cbfn_fall);
   

2. Record for one or more PWM periods of time to record enough duty count.

   wait_ms((NUM_TESTS + 1)*period_in_ms);
   

   In duty_cbfn_fall, make sure correct duty count is recorded.

   if (duty_fall_count == NUM_TESTS) {
       return;
   }
   if(last_rise_time == 0){
       // do nothing
   } else{
       duty_fall_count ++;
       duty_running_count = duty_running_count + (duty_timer.read_ms() - last_rise_time);
   }
   

Issues / proposals for handling NC Pins

1 - If a pin resolves to a no-connect "NC", then the test should fail quickly. Every test should start with an assert that checks that pins under test are not NC.

2 - Tests such as "tests-api-analogin" do not fail quickly when the pin under test is NC. It waits to time out and results in an Error and long wasted test time. It would be more useful if the test result was "NC" instead of "FAIL" or "ERROR"

3- It is unavoidable for some platforms to have some pins not connected (NC). This information is critical to users. It would be great if the CI Test Shield test end results also generated a Table with a list of the pins tested and the result, including if they are NC pins.

MBED_CONF_APP_DIO_X not well named for PWM tests

I just noticed that MBED_CONF_APP_DIO_X have been added to be used for PWM tests and others.

MBED_CONF_APP_DIO_2 means D2, it makes no sense to re-redefine MBED_CONF_APP_DIO_2 to something else I think, ... clearly I have mis-understood something.

In case of PWM, what we need is to defined a pair PWM-OUT / PWM-IN
where
PWM-OUT is a valid IO for a pwm (MBED_CONF_APP_PWM_0 used for this purpose)
PWM-IN is the IO that is wired with a resistor to PWM-OUT IO on the CI shield

so rather than MBED_CONF_APP_DIO_X, I would propose to use something like MBED_CONF_APP_PWM_0_INT
MBED_CONF_APP_PWM_1_INT
MBED_CONF_APP_PWM_2_INT
MBED_CONF_APP_PWM_3_INT
meaning input interrupt to check PWM_0

this would avoid dependency between PWM tests and BusInOut / InterruptIn tests

Consider adding a time keeping device

In mbed OS testing, we currently struggle to have a trusted time keeping source to compare the hardware timers. We have been using the host PC as the trusted time source, but it isn't a real time system so the time can skew significantly by the time it is communicated with the device.

If we could have something like an RTC or a 555 timer circuit as a trusted time source, it would allow us to more accurately measure time and compare it with the device-reported time.

I2CEeprom library update

I have reported an issue in I2CEeprom library and proposed an update in the below PR:
https://developer.mbed.org/users/rhourahane/code/I2CEeprom/pull-request/1

But I haven't received an answer yet.
An alternative is to use a fork of this library (mine or another one from ci-test team where the pull requests can be submitted for review, discussions and merges.

I2C Flash BOM Update

Suggested update: update I2C Flash chip to 512KB version (http://www.digikey.com/product-detail/en/microchip-technology/24LC512-I-SN/24LC512-I-SN-ND/2270915 ) so it can be used with the FS Tests. current one is too small.

It is identical, should be drop in replacement.

Priority: nice to have

AnalogIn test Failed on a new platform because the voltage cannot drop so quick

I got a AnalogIn test failure.

Log analysis

1. Following log shows 1st call of AnalogInput_Test(), which located in ci-test-shield\TESTS\API\AnalogIn\AnalogIn.cpp, passed, then followed calls (case 2, ..., case 5) got failed.
2. All failures hit TEST_ASSERT_MESSAGE(ain.read() > prev_value,"Analog Input did not increment. Check that you have assigned valid pins in mbed_app.json file") at 1st loop in for(x = 0; x<5; x++) in AnalogInput_Test().
3. Which means that at beginning of loops, the AnalogIn value with outputs = 0, which cleared all dout_pin bits (= turned off the output voltage), is bigger than the AnalogIn value with outputs =1, which only set the least dout_pin bit.

A. The log with more printf, the failed point is marked by <<

[1500546174.03][CONN][RXD] >>> Running case #1: 'Analog Input on AIN_0'...
[1500546174.08][CONN][INF] found KV pair in stream: {{__testcase_start;Analog Input on AIN_0}}, queued...
[1500546174.09][CONN][RXD] outputs=0x1
[1500546174.11][CONN][RXD] prevValue=0.000000
[1500546174.13][CONN][RXD] ain=0.015640
[1500546174.13][CONN][RXD]
[1500546174.13][CONN][RXD] outputs=0x3
[1500546174.16][CONN][RXD] prevValue=0.027370
[1500546174.18][CONN][RXD] ain=0.043011
[1500546174.18][CONN][RXD]
[1500546174.18][CONN][RXD] outputs=0x7
[1500546174.21][CONN][RXD] prevValue=0.055718
[1500546174.23][CONN][RXD] ain=0.070381
[1500546174.23][CONN][RXD]
[1500546174.23][CONN][RXD] outputs=0xf
[1500546174.25][CONN][RXD] prevValue=0.083089
[1500546174.26][CONN][RXD] ain=0.097752
[1500546174.26][CONN][RXD]
[1500546174.28][CONN][RXD] outputs=0x1f
[1500546174.31][CONN][RXD] prevValue=0.111437
[1500546174.31][CONN][RXD] ain=0.125122
[1500546174.33][CONN][RXD]
[1500546174.36][CONN][INF] found KV pair in stream: {{__testcase_finish;Analog Input on AIN_0;1;0}}, queued...
[1500546174.43][CONN][RXD] >>> 'Analog Input on AIN_0': 1 passed, 0 failed
[1500546174.43][CONN][RXD]
[1500546174.48][CONN][RXD] >>> Running case #2: 'Analog Input on AIN_1'...
[1500546174.53][CONN][RXD] outputs=0x1
[1500546174.53][CONN][INF] found KV pair in stream: {{__testcase_start;Analog Input on AIN_1}}, queued...
[1500546174.55][CONN][RXD] prevValue=0.160313 << in 2nd call of AnalogInput_Test()
[1500546174.56][CONN][RXD] ain=0.074291
[1500546174.56][CONN][RXD]
[1500546174.68][CONN][RXD] :50::FAIL: Analog Input did not increment. Check that you have assigned valid pins in mbed_app.json file
[1500546174.69][CONN][RXD] outputs=0x3
[1500546174.71][CONN][RXD] prevValue=0.027370
[1500546174.73][CONN][RXD] ain=0.043011
[1500546174.73][CONN][RXD]
[1500546174.73][CONN][RXD] outputs=0x7
[1500546174.76][CONN][RXD] prevValue=0.055718
[1500546174.78][CONN][RXD] ain=0.070381
[1500546174.78][CONN][RXD]
[1500546174.78][CONN][RXD] outputs=0xf
[1500546174.81][CONN][RXD] prevValue=0.083089
[1500546174.83][CONN][RXD] ain=0.097752
[1500546174.83][CONN][RXD]
[1500546174.83][CONN][RXD] outputs=0x1f
[1500546174.86][CONN][RXD] prevValue=0.111437
[1500546174.86][CONN][RXD] ain=0.125122
[1500546174.86][CONN][RXD]
[1500546174.93][CONN][INF] found KV pair in stream: {{__testcase_finish;Analog Input on AIN_1;0;1}}, queued...
[1500546175.01][CONN][RXD] >>> 'Analog Input on AIN_1': 0 passed, 1 failed with reason 'Test Cases Failed'
[1500546175.01][CONN][RXD]
[1500546175.06][CONN][RXD] >>> Running case #3: 'Analog Input on AIN_2'...
[1500546175.11][CONN][INF] found KV pair in stream: {{__testcase_start;Analog Input on AIN_2}}, queued...
[1500546175.13][CONN][RXD] outputs=0x1
[1500546175.13][CONN][RXD] prevValue=0.155425 << in 3rd call of AnalogInput_Test()
[1500546175.15][CONN][RXD] ain=0.072336
[1500546175.15][CONN][RXD]
[1500546175.26][CONN][RXD] :50::FAIL: Analog Input did not increment. Check that you have assigned valid pins in mbed_app.json file
[1500546175.28][CONN][RXD] outputs=0x3
[1500546175.30][CONN][RXD] prevValue=0.027370
[1500546175.31][CONN][RXD] ain=0.042033
[1500546175.31][CONN][RXD]
[1500546175.33][CONN][RXD] outputs=0x7
[1500546175.35][CONN][RXD] prevValue=0.055718
[1500546175.36][CONN][RXD] ain=0.069404
[1500546175.36][CONN][RXD]
[1500546175.38][CONN][RXD] outputs=0xf
[1500546175.39][CONN][RXD] prevValue=0.083089
[1500546175.41][CONN][RXD] ain=0.097752
[1500546175.41][CONN][RXD]

Fax plan A & B

4. when the 1st call of following function is passed, the voltage is high. then 2nd call happened.
   dout_pin bits are cleared at line a, at this moment the voltage is still high cannot drop quickly.
5. Plan A
   A 100ms delay loop is added at line b. to wait for the voltage going down. If I remove the line b, and run to line c, at this moment x=0, then prev_value will be a big value > followed ain.read() to hit the TEST_ASSERT_MESSAGE().
   but I'm not sure how many targets will have this voltage dropping slow issue. it seems like that the most of targets don't need the 100ms wait to run AnalogIn test case. so I made another fix.
6. Plan B
   It's not necessary to add line b delay loop, but line d.
   Because the root cause is that the voltage output is not turned off at end of previous test, so I just add line d to stop voltage output at end of AnalogInput_Test().

B. ci-test-shield\TESTS\API\AnalogIn\AnalogIn.cpp, with fixed code

// Template to set one Analog pin as input and then cycle through the rest as outputs.
// As you turn more pins on the voltage on the ain pin will go up.
template <PinName ain_pin, PinName dout_pin1, PinName dout_pin2, PinName dout_pin3, PinName dout_pin4, PinName dout_pin5> 
void AnalogInput_Test()
{
    AnalogIn ain(ain_pin);
    BusInOut outputs(dout_pin1,dout_pin2,dout_pin3,dout_pin4,dout_pin5);
    outputs.output();
    int x = 0;
    int y= 0;
    outputs = 0;   //line a
    while(ain.read()>0.0f){    //line b, actually the loop only run once on xxxM066
        wait(0.1);    // plan A : the effective delay is required for xxxM066. 7/12 - ren
    }
    float prev_value = 0;
    for(x = 0; x<5; x++) {
        DEBUG_PRINTF("X=%d\n",x);
        prev_value = ain.read();    // line c, the 1st read value, when x=0, should be 0.0f !
        y = (y<<1) + 1;
        outputs = y;
        //wait(0.1);      // this delay is not effective, cannot fix the failures. 7/12 - ren
        DEBUG_PRINTF("outputs=0x%x\nprevValue=%f\nain=%f\n\n",y,prev_value,ain.read());
        printf("outputs=0x%x\nprevValue=%f\nain=%f\n\n",y,prev_value,ain.read());
        TEST_ASSERT_MESSAGE(ain.read() > prev_value,"Analog Input did not increment. Check that you have assigned valid pins in mbed_app.json file")
    }
    DEBUG_PRINTF("Finished the Test\n");
    outputs = 0;        //line d, plan B : 2nd way to fix, this is better. ren
    TEST_ASSERT(true);
}

I2CEeprom ARMC6 compilation failed with mbed-os master

Hi

Due to new MBED compilation options, I2CEeprom can't be compiled any more...

[Error] I2CEeprom.cpp@32,23: non-constant-expression cannot be narrowed from type 'std::size_t' (aka 'unsigned int') to 'char' in initializer list [-Wc++11-narrowing]
[Error] I2CEeprom.cpp@32,39: non-constant-expression cannot be narrowed from type 'unsigned int' to 'char' in initializer list [-Wc++11-narrowing]
[Error] I2CEeprom.cpp@47,23: non-constant-expression cannot be narrowed from type 'std::size_t' (aka 'unsigned int') to 'char' in initializer list [-Wc++11-narrowing]
[Error] I2CEeprom.cpp@47,39: non-constant-expression cannot be narrowed from type 'unsigned int' to 'char' in initializer list [-Wc++11-narrowing]
[Error] I2CEeprom.cpp@62,23: non-constant-expression cannot be narrowed from type 'std::size_t' (aka 'unsigned int') to 'char' in initializer list [-Wc++11-narrowing]
[Error] I2CEeprom.cpp@62,39: non-constant-expression cannot be narrowed from type 'unsigned int' to 'char' in initializer list [-Wc++11-narrowing]
[Error] I2CEeprom.cpp@101,27: non-constant-expression cannot be narrowed from type 'std::size_t' (aka 'unsigned int') to 'char' in initializer list [-Wc++11-narrowing]
[Error] I2CEeprom.cpp@101,43: non-constant-expression cannot be narrowed from type 'unsigned int' to 'char' in initializer list [-Wc++11-narrowing]
[Error] I2CEeprom.cpp@161,27: non-constant-expression cannot be narrowed from type 'std::size_t' (aka 'unsigned int') to 'char' in initializer list [-Wc++11-narrowing]
[Error] I2CEeprom.cpp@161,43: non-constant-expression cannot be narrowed from type 'unsigned int' to 'char' in initializer list [-Wc++11-narrowing]

Meet tests-api-analogin bug?

SHA
53d4476

Description
I am testing ci-test-shield on NuMaker-PFM-M453 board and meet a potential bug in tests-api-analogin. In TESTS/API/AnalogIn/AnalogIn.cpp > AnalogInput_Test,

    AnalogIn ain(ain_pin);
    BusInOut outputs(dout_pin1,dout_pin2,dout_pin3,dout_pin4,dout_pin5);
    outputs.output();
    int x = 0;
    int y= 0;
    outputs = 1;
    float prev_value = 0; 

The outputs bus should initialize to 0:

    AnalogIn ain(ain_pin);
    BusInOut outputs(dout_pin1,dout_pin2,dout_pin3,dout_pin4,dout_pin5);
    outputs.output();
    int x = 0;
    int y= 0;
    outputs = 0;
    float prev_value = 0; 

Missing prerequisites for running ci test shield tests

Need to add to the list of Pre-requisites

• git
• mercurial
• compiler such as GCC, ARM Compiler 5, or IAR

also python modules

• mbed-ls
• pyserial

For the python modules, how about a requirements file that can ensure they are installed? Suggest to add a requirements.txt file.
And add this command to the setup instructions:
pip install -r requirements.txt

Tests fail to compile if platform doesn't support PWM

Target: ARM_BEETLE_SOC
Toolchain: ARMCC5

Build failures:
  * ARM_BEETLE_SOC::ARM::TESTS-ASSUMPTIONS-PWMOUT
        Building project PwmOut (ARM_BEETLE_SOC, ARM)
        Scan: ARM
        Scan: FEATURE_BLE
        Scan: PwmOut
        [DEBUG] Macros: -D__MBED__=1 -DTARGET_LIKE_MBED -DWSF_TOKEN_ENABLED=FALSE -D__MBED_CMSIS_RTOS_CM -DDEVICE_LOWPOWERTIMER=1 -DTOOLCHAIN_object -D__CMSIS_RTOS -DDEVICE_CLCD=1 -DWSF_MAX_HANDLERS=20 -DWSF_MS_PER_TICK=20 -DMBED_BUILD_TIMESTAMP=1476466707.37 -DNO_LEDS -DTARGET_LIKE_CORTEX_M3 -DTARGET_CORTEX_M -DASIC -DARM_MATH_CM3 -DTARGET_UVISOR_UNSUPPORTED -DFEATURE_BLE=1 -DTARGET_M3 -DCONFIG_HOST_REV=0x20 -DCONFIG_ALLOW_SETTING_WRITE=TRUE -DDEVICE_INTERRUPTIN=1 -DDEVICE_I2C=1 -DDEVICE_PORTOUT=1 -D__CORTEX_M3 -DCONFIG_ALLOW_DEEP_SLEEP=FALSE -DWSF_TRACE_ENABLED=TRUE -DTARGET_RELEASE -DDEVICE_PORTINOUT=1 -DWSF_ASSERT_ENABLED=FALSE -DHCI_VS_TARGET -DTARGET_BEETLE -DDEVICE_PORTIN=1 -DDEVICE_SLEEP=1 -DTARGET_ARM_SSG -DTARGET_ARM_BEETLE_SOC -DDEVICE_SPI=1 -DWSF_PRINTF_MAX_LEN=128 -DDEVICE_ANALOGIN=1 -DDEVICE_SERIAL=1 -DCMSDK_BEETLE -DTOOLCHAIN_ARM -DTOOLCHAIN_ARM_STD
        Compile [100.0%]: PwmOut.cpp
        [DEBUG] Compile: C:\Program Files\ARM\bin\armcc -c --gnu -Otime --split_sections --apcs=interwork --brief_diagnostics --restrict --multibyte_chars -O3 --cpu=Cortex-M3 --md --no_depend_system_headers --c99 -D__ASSERT_MSG --cpp --no_rtti --no_vla -I "C:\Program Files\ARM\include" -D__MBED__=1 -DTARGET_LIKE_MBED -DWSF_TOKEN_ENABLED=FALSE -D__MBED_CMSIS_RTOS_CM -DDEVICE_LOWPOWERTIMER=1 -DTOOLCHAIN_object -D__CMSIS_RTOS -DDEVICE_CLCD=1 -DWSF_MAX_HANDLERS=20 -DWSF_MS_PER_TICK=20 -DMBED_BUILD_TIMESTAMP=1476466707.37 -DNO_LEDS -DTARGET_LIKE_CORTEX_M3 -DTARGET_CORTEX_M -DASIC -DARM_MATH_CM3 -DTARGET_UVISOR_UNSUPPORTED -DFEATURE_BLE=1 -DTARGET_M3 -DCONFIG_HOST_REV=0x20 -DCONFIG_ALLOW_SETTING_WRITE=TRUE -DDEVICE_INTERRUPTIN=1 -DDEVICE_I2C=1 -DDEVICE_PORTOUT=1 -D__CORTEX_M3 -DCONFIG_ALLOW_DEEP_SLEEP=FALSE -DWSF_TRACE_ENABLED=TRUE -DTARGET_RELEASE -DDEVICE_PORTINOUT=1 -DWSF_ASSERT_ENABLED=FALSE -DHCI_VS_TARGET -DTARGET_BEETLE -DDEVICE_PORTIN=1 -DDEVICE_SLEEP=1 -DTARGET_ARM_SSG -DTARGET_ARM_BEETLE_SOC -DDEVICE_SPI=1 -DWSF_PRINTF_MAX_LEN=128 -DDEVICE_ANALOGIN=1 -DDEVICE_SERIAL=1 -DCMSDK_BEETLE -DTOOLCHAIN_ARM -DTOOLCHAIN_ARM_STD --via C:\mbed\ci-test-shield\BUILD\tests\ARM_BEETLE_SOC\ARM\.\TESTS\assumptions\PwmOut\.includes_9ab35f1cd05821957a49a5747d99cc80.txt --preinclude=C:\mbed\ci-test-shield\BUILD\tests\ARM_BEETLE_SOC\ARM\.\TESTS\assumptions\PwmOut\mbed_config.h --depend C:\mbed\ci-test-shield\BUILD\tests\ARM_BEETLE_SOC\ARM\.\TESTS\assumptions\PwmOut\.\PwmOut.d -o C:\mbed\ci-test-shield\BUILD\tests\ARM_BEETLE_SOC\ARM\.\TESTS\assumptions\PwmOut\.\PwmOut.o .\TESTS\assumptions\PwmOut\PwmOut.cpp
        [Error] PwmOut.cpp@4,0:  #35: #error directive: PWMOUT not supported on this platform, add 'DEVICE_PWMOUT' deffinition to your platform.
        [DEBUG] Return: 1
        [DEBUG] Output: ".\TESTS\assumptions\PwmOut\PwmOut.cpp", line 4: Error:  #35: #error directive: PWMOUT not supported on this platform, add 'DEVICE_PWMOUT' deffinition to your platform.

        [DEBUG] Output: .\TESTS\assumptions\PwmOut\PwmOut.cpp: 0 warnings, 1 error

This prevents further testing

Prepare repository for production releases

Make master the latest release. Only commits tag with a release tag should be pulled into master.

development will be for current development where feature branches will be merged.

level-testing planned for Q3 as a feature branch

I will be making these changes later this week

SPI - test for hw SSEL

SPI test currently is using SD card, that uses by default sw SSEL. We should define a test case for hw ssel.

Related issue to #21 (changing hw ssel to sw ssel)l

mBed-HDK link has moved.

Should now point to CITestShield

Speed Up Tests

Problem

Some tests take way too long to run, like PWM, some of those take ~10s

Solution

Add #error like functionality so if part way through the test a fail is obvious then it exits and moves on.

Some greentea warnings about summary miscount for assumption tests

Target: All
Toolchain: ARMCC5

mbedgt: checking for GCOV data...
mbedgt: mbed-host-test-runner: stopped and returned 'OK'
mbedgt: test on hardware with target id: 07770221075469163810FF5C
mbedgt: test suite 'tests-assumptions-analogin' ...................................................... OK in 10.82 sec
        test case: 'AnalogIn - can co-exist with DigitalOut' ......................................... OK in 0.07 sec
        test case: 'AnalogIn - existence of `DEVICE_ANALOGIN` macro' ................................. OK in 0.08 sec
        test case: 'AnalogIn - pin can do DigitalOut' ................................................ OK in 0.07 sec
mbedgt: test case summary: 8 passes, 0 failures
mbedgt: utest test case summary mismatch: utest reported passes and failures miscount!
        reported by utest: passes = 8, failures 0)
        test case result count: passes = 3, failures 0)

Update I2C Driver to BlockDevice Driver

Suggestion

Update the I2C driver to use https://github.com/armmbed/i2ceeprom-driver instead as this is maintained by the ARM mbed team and has support for BlockDevice which enables the File System driver.

Priority : Low

This is not necessary, but should be considered long term.

Test cases - the summary

This issue is to track what this CI shield test repo covers and extend it. I would like to keep this updated ,thus take this as a first draft and based on the feedback we update it as needed, and start implementing additions/changes.

1. AnalogIn

    Test cases

    - Analog create and destroy AnalogIn object
    - Analog Input on AIN0 - read() + read_u16()
    - Analog Input on AIN1 - read() + read_u16()
    - Analog Input on AIN2 - read() + read_u16()
    - Analog Input on AIN3 - read() + read_u16()
    - Analog Input on AIN4 - read() + read_u16()
    - Analog Input on AIN5 - read() + read_u16()

2. AnalogOut

    Test cases

    - AnalogOut create and destroy AnalogOut object
    - AnalogOut on AOUT - read(), write(), write_u16()

3. BusIn/Out/InOut

    Test cases

    - BusIn, BusOut and BusInOout create and destroy objects
    - BusInOut to BusInOut - read and write, input and output
    - BusIn - mode() - pullup/down

4. Digital IO

    Test cases

    - DigitalIn/Out create and destroy objects
    - DigitalIn, Out - read/write
    - mode()
    - is_connected()

5. I2C

    Test cases

    // all below are using the default frequency
    - create and destroy I2C object 
    - Invalid write/read (to check return values for failures)
    - LM75B temperature read
    - EEPROM WR single byte
    - EEPROM WR 2 bytes
    - EEPROM WR 10 bytes
    - EEPROM WR 100 bytes

6. InterruptIn

    Test cases

    - create and destroy InterruptIn object
    - Register callback for falling edge
    - Register callback for raising edge
    - enable and disable IRQ
    - read() values

7. PWM

    Test cases

    - create and destroy PWM object
    - Frequency 10ms, 30, 100 and 500ms on all PWM pins available
    - Duty cycle 10, 30, 100, 500ms on all PWM pins available

8. SPI

    Test cases

    // assumes the default frequency
    - create and destroy SPI object
    - open filehandle for SD card
    - write to the SD card
    - read from the SD card

SD SPI test not working with mbed-os/master

Since FATFileSystem was introduced in mbed-os, the ci tests aren't compiling anymore because of redefinition of symbols.

    Error: L6200E: Symbol ff_convert multiply defined (by ccsbcs.o and ccsbcs.o).
    Error: L6200E: Symbol ff_wtoupper multiply defined (by ccsbcs.o and ccsbcs.o).
    Error: L6200E: Symbol disk_initialize multiply defined (by FATFileSystem.o and diskio.o).
    Error: L6200E: Symbol disk_ioctl multiply defined (by FATFileSystem.o and diskio.o).
    Error: L6200E: Symbol disk_read multiply defined (by FATFileSystem.o and diskio.o).
    Error: L6200E: Symbol disk_status multiply defined (by FATFileSystem.o and diskio.o).
    Error: L6200E: Symbol disk_write multiply defined (by FATFileSystem.o and diskio.o).
    Error: L6200E: Symbol clust2sect(FATFS*, unsigned long) multiply defined (by ff.o and ff.o).
    Error: L6200E: Symbol get_fat(FATFS*, unsigned long) multiply defined (by ff.o and ff.o).
    Error: L6200E: Symbol put_fat(FATFS*, unsigned long, unsigned long) multiply defined (by ff.o and ff.o).
    Error: L6200E: Symbol f_chmod multiply defined (by ff.o and ff.o).
    Error: L6200E: Symbol f_close multiply defined (by ff.o and ff.o).
    Error: L6200E: Symbol f_closedir multiply defined (by ff.o and ff.o).
    Error: L6200E: Symbol f_getfree multiply defined (by ff.o and ff.o).
    Error: L6200E: Symbol f_lseek multiply defined (by ff.o and ff.o).
    Error: L6200E: Symbol f_mkdir multiply defined (by ff.o and ff.o).
    Error: L6200E: Symbol f_mkfs multiply defined (by ff.o and ff.o).
    Error: L6200E: Symbol f_mount multiply defined (by ff.o and ff.o).
    Error: L6200E: Symbol f_open multiply defined (by ff.o and ff.o).
    Error: L6200E: Symbol f_opendir multiply defined (by ff.o and ff.o).
    Error: L6200E: Symbol f_read multiply defined (by ff.o and ff.o).
    Error: L6200E: Symbol f_readdir multiply defined (by ff.o and ff.o).
    Error: L6200E: Symbol f_rename multiply defined (by ff.o and ff.o).
    Error: L6200E: Symbol f_stat multiply defined (by ff.o and ff.o).
    Error: L6200E: Symbol f_sync multiply defined (by ff.o and ff.o).
    Error: L6200E: Symbol f_truncate multiply defined (by ff.o and ff.o).
    Error: L6200E: Symbol f_unlink multiply defined (by ff.o and ff.o).
    Error: L6200E: Symbol f_utime multiply defined (by ff.o and ff.o).
    Error: L6200E: Symbol f_write multiply defined (by ff.o and ff.o).
    Error: L6200E: Symbol get_fattime multiply defined (by FATFileSystem.o and FATFileSystem.o).
    Error: L6200E: Symbol FATFileSystem::mkdir(const char*, int) multiply defined (by FATFileSystem.o and FATFileSystem.o).
    Error: L6200E: Symbol FATFileSystem::remove(const char*) multiply defined (by FATFileSystem.o and FATFileSystem.o).
    Error: L6200E: Symbol FATFileSystem::rename(const char*, const char*) multiply defined (by FATFileSystem.o and FATFileSystem.o).
    Error: L6200E: Symbol FATFileSystem::unmount() multiply defined (by FATFileSystem.o and FATFileSystem.o).
    Error: L6200E: Symbol FATFileSystem::~FATFileSystem__deallocating() multiply defined (by FATFileSystem.o and FATFileSystem.o).
    Error: L6200E: Symbol FATFileSystem::~FATFileSystem() multiply defined (by FATFileSystem.o and FATFileSystem.o).
    Error: L6200E: Symbol FATFileSystem::~FATFileSystem__sub_object() multiply defined (by FATFileSystem.o and FATFileSystem.o).
    Not enough information to list the image map.
    Finished: 1 information, 0 warning and 37 error messages.

NCS36510 target overrides not released

The target NCS36510 requires some overrides but they are not present in the current stable release.

Here is what was added:

        "NCS36510": {
            "AOUT": "NC",
            "DIO_7":"NC",
            "DIO_8":"NC",
            "DIO_9":"NC",
            "AIN_4": "NC",
            "AIN_5": "NC"
        },        

@maclobdell

Feature: Simulatenous drivers testing

It would be great if the CI test shield tests are verifying the simultaneous use of mbed OS driver APIs:

• Low level GPIO - DigitalOut, DigitalIn, InterruptIn, AnalogIn
• Comm APIs - I2C + SPI simultaneous use of EEPROM/I2C and SDCard/SPI.
  ** Simultaneous use in a single thread
  ** Simultaneous use in multiple threads
• Mixed - GPIO & Comm APIs - both for single and multi-threaded scenarios

Printing the tests output would effectively test the simultaneous use of UART as well.

@BlackstoneEngineering @mbartling @bridadan @sg-

DELTA_DFCM_NNN50 can not bypass pins A4, A5 in AnalogIn test case

Currently, pins A4 and A5 of DELTA_DFCM_NNN50 are used for external RC circuit.
While doing CI test, AnalogIn.cpp test throughout A0 to A5 by default result in ERROR on DELTA_DFCM_NNN50
Could you create a option said AIN_4_OUT and AIN_5_OUT in order to override in mbed_app.json?

Large number of failures a result of trying to test/use too many pins

I've been testing many platforms lately...with lots of failures.

A common issue is that not all platforms connect each pin on the Arduino header.

For example, LPC546XX doesn't have D6, D7, A4, A5 connected by default.

If I want to pass tests that relay on D6, D7, A4, A5, as digital outs (e.g. Bus tests) then I have to find surrogate pins to fly-wire to those locations.

The problem is that all other pins are taken!
D2-D9 (Digital, bus , interrupt tests)
D10-D13(SPI, bus tests)
D14-D15 (I2C, bus tests)
A0-A3(Digital, Analog, Bus tests)

If the tests attempted to test less pins, say just 2-3 per peripheral type, then there would be a higher chance of finding alternate pins to wire in.

My guess is that we would find 95% of issues by just testing two pins of each type.

This would reduce the complexity, make this project much more stable, and widely used.

The only drawback I can see is that we wouldn't be catching and highlighting deviations from the Arduiono standard, but honestly, I don't think that is the job of Mbed OS.

Avoid wrong AnalogOut test

Hi

tests-api-analogout test is checking if device support DEVICE_ANALOGOUT,
but it supposes then, that this DAC pin is between A0 and A5...
which it is not always the case ...?

Maybe test should check first that AOUT pin is between A0 and A5 ?
Goal is just to avoid the "ERROR" result.

Thx

Add async tests

We got some async test in the mbed-os, that shall be added/or rewrite or write a new one.

To be added:

• SPI
• I2C
• serial

Tests references:

• https://github.com/ARMmbed/mbed-os/blob/master/targets/TARGET_Silicon_Labs/TARGET_EFM32/TESTS/spi/basic-spi/main.cpp
• https://github.com/ARMmbed/mbed-os/tree/master/features/unsupported/tests/utest (async tests)

Docs - add explanation of difference between FAIL and ERROR cases

It has come to my attention that we do not clearly define the difference between a failure and an error in the results. Need to add explanation about what each case is and how to best debug.

I2C Test assert not catching real failures

The I2C EEPROM tests should fail on K64F for an other issue. Basically, due to this other issue, the K64F isn't able to read from the EEPROM. But, the ci-test-shield I2C EEPROM tests for writing/reading 10 and 100 bytes are showing OK. This is not correct.

Here is the very verbose (--vv) log for one of the tests. It shows that the number of bytes read is 0 and the Read string is null. But the asserts are not catching this.

[1481314762.72][CONN][RXD] >>> Running case #5: 'I2C -  EEProm WR 10  Bytes'...
[1481314762.77][CONN][RXD]
[1481314762.77][CONN][INF] found KV pair in stream: {{__testcase_start;I2C -  EEProm WR 10  Bytes}}, queued...
[1481314762.78][CONN][RXD] ****
[1481314762.82][CONN][RXD] Buffer Len = `10`, String = `DYOQYKFDB`
[1481314762.83][CONN][RXD] ****
[1481314762.83][CONN][RXD]
[1481314762.83][CONN][RXD] ****
[1481314762.86][CONN][RXD]  Test String = `DYOQYKFDB`
[1481314762.87][CONN][RXD] ****
[1481314762.90][CONN][RXD] num_read: 0, num_written: 10
[1481314762.90][CONN][RXD]
[1481314762.91][CONN][RXD] ****
[1481314762.93][CONN][RXD]  Address = `100`
[1481314762.94][CONN][RXD]  Len = `10`
[1481314762.97][CONN][RXD]  Num Bytes Written = `10`
[1481314762.99][CONN][RXD]  Num Bytes Read = `0`
[1481314763.03][CONN][RXD]  Written String = `DYOQYKFDB`
[1481314763.05][CONN][RXD]  Read String = ``
[1481314763.05][CONN][RXD] ****
[1481314763.11][CONN][INF] found KV pair in stream: {{__testcase_finish;I2C -  EEProm WR 10  Bytes;1;0}}, queued...
[1481314763.16][CONN][RXD] >>> 'I2C -  EEProm WR 10  Bytes': 1 passed, 0 failed

+--------------+---------------+---------------+------------------------------------+--------+--------+--------+--------------------+
| target       | platform_name | test suite    | test case                          | passed | failed | result | elapsed_time (sec) |
+--------------+---------------+---------------+------------------------------------+--------+--------+--------+--------------------+
| K64F-GCC_ARM | K64F          | tests-api-i2c | I2C -  EEProm WR 10  Bytes         | 1      | 0      | OK     | 0.34               |
| K64F-GCC_ARM | K64F          | tests-api-i2c | I2C -  EEProm WR 100 Bytes         | 1      | 0      | OK     | 0.74               |
| K64F-GCC_ARM | K64F          | tests-api-i2c | I2C -  EEProm WR 2 Bytes           | 0      | 1      | FAIL   | 0.35               |
| K64F-GCC_ARM | K64F          | tests-api-i2c | I2C -  EEProm WR Single Byte       | 0      | 2      | FAIL   | 0.31               |
| K64F-GCC_ARM | K64F          | tests-api-i2c | I2C -  Instantiation of I2C Object | 1      | 0      | OK     | 0.06               |
| K64F-GCC_ARM | K64F          | tests-api-i2c | I2C -  LM75B Temperature Read      | 1      | 0      | OK     | 0.12               |
+--------------+---------------+---------------+------------------------------------+--------+--------+--------+--------------------+

It looks like one of the assert types is distorting the results from the other asserts.

If these two lines are commented out in void flash_WR(), then the tests result in the expected and correct behavior (fail when it actually fails). However, with these lines not-commented out, the tests result in OK.

    TEST_ASSERT_MESSAGE(strcmp((char *)test_string,(char *)read_string) == 0,"String Written != String Read");
    TEST_ASSERT_MESSAGE(strcmp((char *)read_string,(char *)test_string) == 0,"String Written != String Read");
    //TEST_ASSERT_EQUAL_STRING_MESSAGE((char *)test_string,(char *)read_string,"String read does not match the string written");
    //TEST_ASSERT_EQUAL_STRING_MESSAGE((char *)read_string,(char *)test_string,"String read does not match the string written");
    TEST_ASSERT_EQUAL_MESSAGE(num_written,num_read,"Number of bytes written does not match the number of bytes read");
    DEBUG_PRINTF("\r\n****\r\n Address = `%d`\r\n Len = `%d`\r\n Num Bytes Written = `%d` \r\n Num Bytes Read = `%d` \r\n Written String = `%s` \r\n Read String = `%s` \r\n****\r\n",address,size_of_data,num_written,num_read,test_string,read_string);

@bridadan mentioned that he thinks this has something to do with the use of STATUS_CONTINUE in greentea_failure_handler

@adbridge
@BlackstoneEngineering

tests-api-spi fails to compile

The SPI tests fails to compile with the latest mbed-os commit ARMmbed/mbed-os@2526ac8.

Need this to test SPI on some of our upcoming platforms

Do not run tests if assumptions fail

Problem

If Assumptions for tests fail then the test should not be run

Solution

Unknown

Correct deprecated API

Lots of deprecated API are used with ci-test-shield

• wait_ms
• wait
• gettid

We need to use this shield even after mbed-os 5.14

Thx

@MarceloSalazar
@LMESTM
@bulislaw

tests-api-analogin : value read max value limitation

Hi

Here is tests-api-analogin scenario :
// As you turn more pins on the voltage on the ain pin will go up.

Note that measurement value is between 0 and VREF+, i.e. 3.3 V in most of ST boards.
My issue is with DISCO_L496AG board : max value is 2.5V

ci-test-shield/TESTS/API/AnalogIn/AnalogIn.cpp

Line 35 in befdcc2

BusInOut outputs(dout_pin1,dout_pin2,dout_pin3,dout_pin4,dout_pin5);

Is it possible to limit the BusOut with 4 pins ?
Test remains quite the same, and it seems that 4 pins, turned on, is OK (below 2.5V) ?

Thx

Reference: ARMmbed/mbed-os#8168

AnalogIn test requires that analog input pins also support digital output function

Need an alternative way to test analog pins when they don't support digital output for some rare cases. A resistor ladder is preferred such that a different value can be read on each analog pin. This would confirm that the analog pins are configured correctly for individually measuring on each pin.
Alternatively, have a way to fly-wire the pwm-capable digital signals (selectable) to a capacitor to ground, then to the resistors in parallel to the analog pins.