AI Project Workflow [Europe_Wiki]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== AI Project Workflow ======

====== INTRODUCTION ======

This document defines the cooperation workflow on AI projects between Streamax and customers. Please read the following content carefully before you work on an AI project or tender as it sets out the terms of necessary actions to ensure the performance.

====== 2 TERMS AND CONDITIONS ======

Before setting off, please fully understand and agree that AI technology product is not the same as other well-defined products, of which features’ performance is with 100% maturity and no need to periodically change. Rather, the biggest advantage of AI technology is with learning ability, the product can be trained by enough footage of actual usage scenarios, to improve the performance better and better.

  * **Accuracy**
      * It’s understandable and reasonable that customers have expectations of AI accuracy rate. We could say that the target can be achieved for DSM and ADAS shortly after training the algorithm, especially for C6D AI, MDVR+AD Kit, AIBOX 4.0.
      * Please note that the algorithm training duration will be longer accordingly for some new AI products.
      * In the beginning, the algorithm may not adapt to local road conditions, in this case, we could use recording metadata provided by our partner to improve the accuracy shortly.
      * If someone irrationally expects the AI system must be 100% accurate, we kindly state that he should not buy AI products. For detailed reasons please refer to the first paragraph.
  * **Definition of fatigue driving**
      * Streamax could detect fatigue driving not only by closed eyes but also by excessive blinking of eyes.
      * Detailed logic: When the vehicle speed is above 20km/h, if it’s detected that the driver closes his eyes for 2/2+ seconds, or blinks 10/10+ times within 1 minute and then closes for 0.5s or 1s.

====== 3 PROJECT WORKFLOW ======

This part is mainly to suggest a better workflow for you, to help with a successful project. Please ensure to align your AI projects info with the Streamax team, especially if you have any new opportunities. Because completed calibration doesn’t mean AI project success, we need to provide support to keep following up with the AI performance and evidence data after kick-off.

===== Stage 1: Preparation =====

Communicate with customers to agree on the expectation, the acceptance should be measurable. Something like, what the customer want is to catch prohibited behaviors as much as possible with tolerable false alarms, or to ensure the accuracy of each alarm with an acceptable false negative. Expectation description should be like this:

  * False-positive alarms of any AI alarm type could be less than 30 per day for 1000 vehicles. Based on our experience in past projects, it’s a challenge to make the false rate lower than 3% as the alarming quantity will be sharply decreased after one month of use.
  * Get as much project information as possible, including but not limit to vehicle model, target installation location, system architecture, detailed verification schedule, sample vehicle quantity, practical installation, and calibration pictures, etc.
  * Communication path
      * A [[internal:draft:wechat|WeChat]] group should be created for each AI project internally and A mail group or Skype group should be created for each AI project with customers.
      * The Streamax support team should include BU, sales team, R&D, especially specific algorithm engineer
      * All of the project info should be passed to the R&D and BU team before POC
  * AI system solution, normally our team will help to choose proper solution.
      * Alarm parameters: adjust parameters based on performance expectation, then export the config file and import it to other MDVRs. Back up the config file.
      * Feature verification test must be completed successfully in the office before being put into practical operation. Ensure to have the same model for future testing.
      * Device model, FMW version should be confirmed by BU and R&D team, any change should be confirmed by BU and R&D
  * Necessary training:
      * Device installation guide and calibration method
      * How to connect indicator to I/O sensor and related setup
      * Troubleshooting procedure
        * Debugging shouldn’t affect MDVR running
        * FMW upgrade method
        * Speed resource
        * Available config file and setup GUI at any time (key point)
  * Checklist:
      * Network status and mobile data
        * Basically speaking, a piece of evidence is about 6M, assuming an MDVR uploads 30 pieces every day and the vehicle runs for 20 days in one month. It turns out that we need 3.6G data for evidence upload, plus basic data upload and recording playback, nearly 8G data is needed (normally it’s two times more than evidence upload’s).
        * What’s the data clearing rule
        * What’s the SIM type? 3G/4G/5G? Is it in VPN?
      * Storage replacement frequency. Please note storage replacement may lead to part of evidence recordings can’t be uploaded to the server in time.
      * Volume setup. Suggest to close voice announcement at the early stage, re-open it after accuracy rate tends to be stable as it may bother driving and cause objection.
  * Reported to CEIBA II and Crocus platform
      * It’s important to count alarm quantities and get the original .264 recording of false alarms for algorithm improvements.
      * Streamax test server will be the best choice.
      * It’s okay if you’d like to only use your own server considering data privacy regulation, but please let us know the login account and password.

===== Stage 2: POC =====

  * Suggest to have a trial running on 1~3 sets, then add sample vehicles gradually according to actual performance
  * ST team need to inform the customer ASAP if notices any false alarm due to incorrect camera installation
  * To track the accuracy rate of false-positive alarms
      * Streamax team will adjust the settings to get all of the alarm recordings, the evidence data will be uploaded to the CEIBA II server. Below is a suggestion about AI evidence uploading.

[[http://15.188.206.201:8800/lib/exe/detail.php?id=internal:draft:ai_project_process&media=internal:draft:023933cb2fc45ed7e930e4dfaa5bd771.png|{{internal:draft:023933cb2fc45ed7e930e4dfaa5bd771.png}}]]

  * If any setup change or FMW upgrade OTA, Streamax needs to inform the customer via e-mail to avoid any misunderstanding on system stability.
  * To track the accuracy rate of false-negative alarms
      * In the POC process, please connect a press button to one of the I/O sensors, if the driver notices any false negatives, please press the button to trigger a period of I/O alarm recording.
      * Please install two more cameras in the whole system, one is facing R-watch, the other one should face the driver to see his behavior.
      * Streamax support team will compare the driver’s actual behavior and the I/O alarm recording to confirm if it’s a real false negative, then take action.
  * Alarm statistics procedure
      * Login to CEIBA II server and click to enter alarm center

[[http://15.188.206.201:8800/lib/exe/detail.php?id=internal:draft:ai_project_process&media=internal:draft:57da76bbcbdbca98c118b70621b95e26.png|{{internal:draft:57da76bbcbdbca98c118b70621b95e26.png}}]]

  * On the ‘Search’ page, tick the target vehicle and choose time range, all alarms will be shown after clicking the ‘search’ button

[[http://15.188.206.201:8800/lib/exe/detail.php?id=internal:draft:ai_project_process&media=internal:draft:582b1af6fda8f210e81f911607eb12a3.png|{{internal:draft:582b1af6fda8f210e81f911607eb12a3.png}}]]

  * Export alarm list, and re-organize the info into statistic template as below "Alarm report 2020.12.08.xlsx"
  * Check if the quantity of evidence is the same as that of alarm. If not, please check the evidence upload strategy then see if some recordings were overwritten before complete uploading. The comparison helps to find out the device abnormality
  * Send the statistic excel and collected false alarm H.264 recordings back to R&D, the accuracy rate is subject to R&D’s statistic result. Any valuable recordings will be used for algorithm training
  * Write down any issue found in the POC process to avoid in the future.
  * We could say that AI projects can be considered to start successfully only when AI performance is good enough after a period of continuous tracking.

===== Stage 3: Project operation =====

Please note that an OTA upgrade is necessary for most AI technology products, so do our competitors. Similarly, Tesla often upgrades their algorithm as well. So we do suggest that every three months or half a year, it would be better to upgrade the algorithm or firmware to get better performance.