Nicla Vision User Manual

Overview

This user manual will guide you through a practical journey covering the most interesting features of the Arduino Nicla Vision. With this user manual, you will learn how to set up, configure and use this Arduino board.

Hardware and Software Requirements

Hardware Requirements

• Arduino Nicla Vision (x1)
• Micro USB cable (x1)

Software Requirements

• OpenMV IDE
• Arduino IDE 1.8.10+, Arduino IDE 2.0+, or Arduino Web Editor
• To create custom Machine Learning models, the Machine Learning Tools add-on integrated into the Arduino Cloud is needed. In case you do not have an Arduino Cloud account, you will need to create one first.

Product Overview

The Nicla Vision is a ready-to-use, standalone camera board for analyzing and processing images on the edge. Thanks to its 2 MP color camera, smart 6-axis motion sensor, integrated microphone, and distance sensor, it is suitable for asset tracking, object recognition, and predictive maintenance.

The Nicla Vision lets you quickly implement sensor nodes to send collected data to the Arduino® Cloud (or third-party vendor services) via its onboard Wi-Fi® and Bluetooth® module.

Board Architecture Overview

The Nicla Vision features a robust and efficient architecture that integrates a range of sensors packed into a tiny footprint. Nicla Vision combines a powerful STM32H747AII6 Dual Arm® Cortex®-M7/M4 IC processor with a 2MP color camera that supports TinyML, as well as a smart 6-axis motion sensor, integrated PDM microphone and a Time of Flight distance sensor.

Here is an overview of main components of the board, as shown in the images above:

• Camera: the Nicla Vision features a camera based on the GC2145 Color rolling shutter image sensor. The GC2145 incorporates a 1616V x 1232H active pixel array, on-chip 10-bit ADC, and an image signal processor. The 2MP GC2145 CMOS camera module is equipped with an 80° (DFOV) stock lens, 1.75 μm pixel size and a focal length of 2.2 mm. It supports RGB output format.
• Microcontroller: the heart of the Nicla Vision is the dual-core STM32H747 (U1), including an Arm® Cortex®-M7 running at 480 MHz and an Arm® Cortex®-M4 running at 240 MHz. The two cores communicate via a Remote Procedure Call mechanism that allows calling functions on the other processor seamlessly.
• Onboard advanced motion sensor: the board features the LSM6DSOX, a smart IMU that includes a 3-axis accelerometer and a 3-axis gyroscope. The LSM6DSOX has a full-scale acceleration range of ±2/±4/±8/±16 g and an angular rate range of ±125/±250/±500/±1000/±2000 dps.
• Onboard distance sensor: the VL53L1CBV0FY Time-of-Flight sensor (U4) adds accurate and low-power ranging capabilities to the Nicla Vision. The invisible near infrared VCSEL laser (including the analog driver) is encapsulated together with receiving optics in an all-in-one small module located below the camera.
• Digital Microphone: the MP34DT05 digital MEMS microphone (U6) is omnidirectional and operates via a capacitive sensing element with a high (64 dB) signal-to-noise ratio. The sensing element, capable of detecting acoustic waves, is manufactured using a specialized silicon micromachining process dedicated to audio sensors production.
• Wireless connectivity: the Murata® LBEE5KL1DX-883 wireless module (U9) simultaneously provides Wi-Fi® and Bluetooth® connectivity in an ultra-small package based on the Cypress CYW4343W. The IEEE802.11 b/g/n Wi-Fi® interface can operate as an access point (AP), station (STA), or dual-mode simultaneous AP/STA. It supports a maximum transfer rate of 65 Mbps. Bluetooth® interface supports Bluetooth® Classic and Bluetooth® Low Energy. An integrated antenna circuitry switch allows a single external antenna (J6) to be shared between Wi-Fi® and Bluetooth®.
• Power management: the Nicla Vision is designed for ultra-low power operation, with efficient power management features that ensure minimal energy consumption even when using always-on motion recognition and image processing. The Nicla Vision features the PF1550 from NXP®, a highly integrated battery charge management integrated circuit (IC) designed for wearables and Internet of Things (IoT) devices.
• Security Elements: the Nicla Vision enables IC level edge-to-cloud security capability through the NXP SE050C2 Crypto chip (U8). This provides Common Criteria EAL 6+ security certification up to OS level, as well as RSA/ECC cryptographic algorithm support and credential storage.

Board Core and Libraries

With OpenMV IDE

Before you can start programming MicroPython scripts for the Nicla Vision, you need to download and install the OpenMV IDE.

Open the OpenMV download page in your browser, download the latest version available for your operating system, and follow the instructions of the installer.

Open the OpenMV IDE and connect the Nicla Vision to your computer via the USB cable if you have not done so yet.

Click on the "connect" symbol at the bottom of the left toolbar.

If your Nicla Vision does not have the latest firmware, a pop-up will ask you to install it. Your board will enter in DFU mode and its green LED will start fading.

Select

Install the latest release firmware

OK

Nicla Vision's green LED will start flashing while the OpenMV firmware is being uploaded to the board. A loading bar will start showing you the flashing progress.

Wait until the green LED stops flashing and fading. You will see a message saying

DFU firmware update complete!

The board will start flashing its blue LED when it is ready to be connected. After confirming the completion dialog, the Nicla Vision should already be connected to the OpenMV IDE, otherwise, click the "connect" button (plug symbol) once again (the blue blinking should stop).

While using the Nicla Vision with OpenMV, the RGB LED of the board can be used to inform the user about its current status. Some of the most important ones are the following:

🟢 Blinking Green: Your Nicla Vision onboard bootloader is running. The onboard bootloader runs for a few seconds when your Nicla Vision is powered via USB to allow OpenMV IDE to reprogram your Nicla Vision.

🔵 Blinking Blue: Your Nicla Vision is running the default main.py script onboard.

If you overwrite the main.py script on your Nicla Vision, then it will run whatever code you loaded on it instead.

If the LED is blinking blue but OpenMV IDE cannot connect to your Nicla Vision, please make sure you are connecting your Nicla Vision to your PC with a USB cable that supplies both data and power.

⚪ Blinking White: Your Nicla Vision firmware is panicking because of a hardware failure. Please check that your Nicla Vision's camera module is installed securely.

If you tap the Nicla Vision reset button once, the board resets. If you tap it twice, the board enters in Device Firmware Upgrade (DFU) mode and its green LED starts blinking and fading.

With Arduino IDE

The Arduino Mbed OS Nicla Boards core contains the libraries and examples you need to work with the board's components, such as its camera and IMU. To install the core for Nicla boards, navigate to Tools > Board > Boards Manager or click the Boards Manager icon in the left tab of the IDE. In the Boards Manager tab, search for

Nicla Vision

Arduino Mbed OS Nicla Boards

To update the bootloader firmware of your Nicla Vision, go to File > Examples > STM32H747_System > STM32H747_manageBootloader and upload this sketch to your board.

After the sketch is uploaded, follow the instructions in the Serial Monitor.

Pinout

The full pinout is available and downloadable as PDF from the link below:

• Nicla Vision full pinout

Datasheet

The complete datasheet is available and downloadable as PDF from the link below:

• Nicla Vision datasheet

Schematics

The complete schematics are available and downloadable as PDF from the link below:

• Nicla Vision schematics

STEP Files

The complete STEP files are available and downloadable from the link below:

• Nicla Vision STEP files

First Use

Powering the Board

The Nicla Vision can be powered by:

• Using a Micro USB cable (not included).
• Using an external 5 V power supply connected to

  VIN

  pin (please, refer to the board pinout section of the user manual).
• Using a 3.7 V Lithium Polymer (Li-Po) battery connected to the board through the onboard battery connector; the manufacturer part number of the battery connector is BM03B-ACHSS and its matching receptacle manufacturer part number is ACHR-03V-S. The recommended minimum battery capacity for the Nicla Vision is 200 mAh. A Li-Po battery with an integrated NTC thermistor is also recommended for thermal protection.
• Using the onboard ESLOV connector, which has a dedicated 5V power line.

Hello World Example

Let's program the Nicla Vision with the classic

hello world

Blink

With OpenMV

Copy and paste the code below into a new sketch in the OpenMV IDE.

Copy
1import time
2from machine import LED
3
4redLED = LED("LED_RED")
5greenLED = LED("LED_GREEN")
6blueLED = LED("LED_BLUE")
7
8while True:
9    redLED.on()
10    time.sleep_ms(1000)
11    redLED.off()
12    time.sleep_ms(1000)
13    greenLED.on()
14    time.sleep_ms(1000)
15    greenLED.off()
16    time.sleep_ms(1000)
17    blueLED.on()
18    time.sleep_ms(1000)
19    blueLED.off()
20    time.sleep_ms(1000)

To run the code on the Nicla Vision, click the Connect button and then the Start button.

With Arduino IDE

Copy and paste the code below into a new sketch in the Arduino IDE.

Copy
1void setup() {
2
3  pinMode(LEDR, OUTPUT);
4  pinMode(LEDG, OUTPUT);
5  pinMode(LEDB, OUTPUT);
6
7  digitalWrite(LEDR, HIGH);
8  digitalWrite(LEDG, HIGH);
9  digitalWrite(LEDB, HIGH);
10
11}
12
13void loop() {
14
15  digitalWrite(LEDR, LOW);  // Nicla Vision LED's turn on with logic '0'
16  delay(1000);                     
17  digitalWrite(LEDR, HIGH);   
18  delay(1000);   
19  digitalWrite(LEDG, LOW);  
20  delay(1000);                     
21  digitalWrite(LEDG, HIGH);   
22  delay(1000);
23  digitalWrite(LEDB, LOW);  
24  delay(1000);                     
25  digitalWrite(LEDB, HIGH);   
26  delay(1000);
27
28}

To upload the code to the Nicla Vision, click the Verify button to compile the sketch and check for errors; then click the Upload button to program the board with the sketch.

Results

You should now repeatedly see the onboard LED turning red, green, and blue.

Pins

Analog Pins

The Nicla Vision has three analog input pins, mapped as follows:

All of them can be used through the built-in functions of the Arduino programming language.

The Nicla Vision ADC reference voltage is fixed to 3.3V, this means that it will map the ADC range from 0 to 3.3 volts.

We will use the Nicla Vision analog inputs on both IDEs, OpenMV and Arduino. For the example codes shown below, we will be reading the analog input

A0

With OpenMV

Using Micropython on the OpenMV IDE the Nicla boards ADC resolution is fixed to 12 bits (it's maximum).

This example code can also be found on File > Examples > Board Control > adc_read_ext_channel.py:

Copy
1import time
2from pyb import ADC
3
4adc = ADC("A0")   # PC4 microcontroller port
5
6while True:
7    # The ADC has 12-bits of resolution for 4096 values.
8    print(adc.read())
9    print("ADC = %fv" % ((adc.read() * 3.3) / 4095))
10    time.sleep_ms(100)

With Arduino IDE

Nicla boards ADC can be configured to 8, 10 or 12 bits defining the argument of the following function respectively (default is 10 bits):

Copy
1analogReadResolution(12);  // ADC resolution set to 12 bits (0-4095)

Example code:

Copy
1int potPin = A0;   // select the input pin for the potentiometer
2
3void setup() {
4  Serial.begin(115200);
5  analogReadResolution(12);
6}
7
8void loop() {
9  // read the value from the sensor:
10  Serial.print("ADC = ");
11  Serial.print((analogRead(potPin) * 3.3) / 4095);;
12  Serial.println(" v");
13  delay(100);
14}

Digital Pins

The Nicla Vision has ten digital pins, mapped as follows:

Notice that I2C and SPI pins can also be used as digital pins. Please, refer to the board pinout section of the user manual to find them on the board.

The analog inputs of the Nicla Vision can be used as digital pins but they can just handle 1.8 V, a greater input may damage the board.

The digital pins of the Nicla Vision can be used as inputs or outputs through the built-in functions of the Arduino programming language.

The Nicla Vision digital I/O's are low power, so to drive output devices like LEDs, resistive loads, buzzers, etc, it is recommended to use a MOSFET driver or a buffer to guarantee the required current flow. Learn more about the Nicla I/O's considerations here.

As an application example after learning the Digital Pins basics, we are going to control an LED using a push button. See the wiring below:

With OpenMV

The configuration of a digital pin is done in the upper section of the code as shown below:

Copy
1# Pin configured as an input
2pin1 = Pin("D1", Pin.IN, Pin.PULL_NONE)          
3# Pin configured as an input, internal pull-up resistor enabled
4pin1 = Pin("D1", Pin.IN, Pin.PULL_UP) 
5# Pin configured as an output
6pin0 = Pin("D0", Pin.OUT_PP, Pin.PULL_NONE)

The pin function can be set as:

Pin.IN

Pin.OUT_PP

Pin.OUT_OD

Pin.AF_PP

Pin.AF_OD

Pin.PULL_NONE

Pin.PULL_UP

Pin.PULL_DOWN

The state of a digital pin, configured as an input, can be read as shown below:

Copy
1# Reads pin1 state, stores value in "state" variable
2state = pin1.value()

The state of a digital pin, configured as an output, can be changed as shown below:

Copy
1# Set pin0 on
2pin0.value(True)    
3# Set pin0 off
4pin0.value(False)

The example code shown below uses digital pin

D0

D1

Copy
1import time
2from machine import Pin
3
4# Define button and LED pin
5button = Pin("D1", Pin.IN, Pin.PULL_UP)
6led = Pin("D0", Pin.OUT_PP, Pin.PULL_NONE)
7
8while True:
9    if button.value() == 0:   # if the button is pressed
10        led.value(1)
11        print("- Button is pressed. LED is on.")
12    else:                     # if the button is not pressed
13        led.value(0)
14        print("- Button is not pressed. LED is off.")
15    time.sleep_ms(1000)       # wait for a second

With Arduino IDE

The configuration of a digital pin is done in the

setup()

pinMode()

Copy
1// Pin configured as an input
2pinMode(D1, INPUT);             
3// Pin configured as an input, internal pull-up resistor enabled
4pinMode(D1, INPUT_PULLUP);  
5// Pin configured as an output
6pinMode(D0, OUTPUT);

The state of a digital pin, configured as an input, can be read using the built-in function

digitalRead()

Copy
1// Reads pin state, stores value in state variable
2state = digitalRead(D1);

The state of a digital pin, configured as an output, can be changed using the built-in function

digitalWrite()

Copy
1// Set pin on
2digitalWrite(D0, HIGH);    
3// Set pin off
4digitalWrite(D0, LOW);

The example code shown below uses digital pin

D0

D1

Copy
1// Define button and LED pin
2int buttonPin = D1;
3int ledPin = D0;
4
5// Variable to store the button state
6int buttonState = 0;
7
8void setup() {
9  // Configure button and LED pins
10  pinMode(buttonPin, INPUT_PULLUP);
11  pinMode(ledPin, OUTPUT);
12  // Initialize Serial communication
13  Serial.begin(115200);
14}
15
16void loop() {
17  // Read the state of the button
18  buttonState = digitalRead(buttonPin);
19  // If the button is pressed, turn on the LED and print its state to the Serial Monitor
20  if (buttonState == LOW) {
21    digitalWrite(ledPin, HIGH);
22    Serial.println("- Button is pressed. LED is on.");
23  } else {
24    // If the button is not pressed, turn off the LED and print to the Serial Monitor
25    digitalWrite(ledPin, LOW);
26    Serial.println("- Button is not pressed. LED is off.");
27  }
28  // Wait for 1000 milliseconds
29  delay(1000);
30}

PWM Pins

Most digital pins of the Nicla Vision can be used as PWM (Pulse Width Modulation) pins, including I2C and SPI interfaces.

With OpenMV

PWM outputs can be controlled with MicroPython easily by using built-in functions as shown below:

First, we need to identify the

Timer

Channel

PWM

Here is a table with the details of the exposed pins on the Nicla Vision:

To use the PWM functions, you need to import the

time

Pin

Timer

Copy
1import time
2from pyb import Pin, Timer

First you need to choose the pin you want to use PWM with.

Copy
1OUT = Pin("D1", Pin.OUT_PP, Pin.PULL_NONE)

Create a timer for the PWM, where you set the

timer number

Copy
1timer1 = Timer(1, freq=1000)

Then you need to start a

PWM channel

Copy
1channel1 = timer1.channel(2, Timer.PWM, pin=OUT, pulse_width_percentage=0)

Get or set the pulse width value on a channel. To get, pass no arguments. To set, give a value as an argument.

Copy
1channel1.pulse_width_percentage(Width) # Width (0-100)

As a complete example here is a code to generate a 50% duty cycle PWM signal at 1 Mhz.

Copy
1import time
2from pyb import Pin, Timer
3
4# D1 is connected to TIMER1_CH2
5
6OUT = Pin("D1")
7
8timer1 = Timer(1, freq=1000000)
9channel1 = timer1.channel(2, Timer.PWM, pin=OUT, pulse_width_percent=0)
10
11channel1.pulse_width_percent(50)
12
13
14while True:
15    time.sleep_ms(1000)

With Arduino IDE

This functionality can be used with the built-in function

analogWrite()

Copy
1analogWrite(pin, value);

The output resolution is 8 bits by default, so the output value should be between 0 and 255. To set a greater resolution, you can use the built-in function

analogWriteResolution

Copy
1analogWriteResolution(bits);

Using

analogWrite

Copy
1// 12 bits PWM 50% duty cycle example code
2
3void setup() {
4  analogWriteResolution(12);    // 12 bits (0-4095)
5}
6
7void loop() {
8  analogWrite(D1, 2048);    // PWM output on D1
9}

Onboard Sensors

The Nicla Vision comes with various onboard sensors that allow you to capture and process motion data via a 6-axis IMU, distance with a Time of Flight (ToF) sensor, record sound with a PDM microphone, and capture images and videos with a camera.

The onboard sensors can be used for developing various applications, such as voice commanded projects, activity recognition, vibration detection and image classification. The onboard sensors are suitable for Machine Learning applications using our Arduino Cloud ML Tools.

IMU

The Nicla Vision features an advanced IMU, which allows the board to sense motion. The IMU on the board is the LSM6DSOXTR from ST®. It consists of a 3-axis accelerometer and a 3-axis gyroscope. They can provide information about the board's motion, orientation, and rotation in a 3D space.

With OpenMV

In this MicroPython environment, you can choose between a basic usage of the IMU by sampling raw motion data using the example code below.

Copy
1import time
2from lsm6dsox import LSM6DSOX
3from machine import Pin
4from machine import SPI
5
6lsm = LSM6DSOX(SPI(5), cs=Pin("PF6", Pin.OUT_PP, Pin.PULL_UP))
7
8while True:
9    print("Accelerometer: x:{:>8.3f} y:{:>8.3f} z:{:>8.3f}".format(*lsm.accel()))
10    print("Gyroscope:     x:{:>8.3f} y:{:>8.3f} z:{:>8.3f}".format(*lsm.gyro()))
11    print("")
12    time.sleep_ms(100)

Also, you can develop Machine Learning applications using the Nicla Vision IMU. As a practical example, we are going to test a

Vibration monitoring

no vibration

low vibration

high vibration

First, download the pre-trained model file from the example repository and copy it to the Nicla Vision storage drive.

Reset the board and run the following code on the OpenMV IDE.

Copy
1from machine import Pin
2from machine import SPI
3from lsm6dsox import LSM6DSOX
4
5INT_MODE = True  # Run in interrupt mode.
6INT_FLAG = False  # Set True on interrupt.
7
8
9def imu_int_handler(pin):
10    global INT_FLAG
11    INT_FLAG = True
12
13
14if INT_MODE is True:
15    int_pin = Pin("PA1", mode=Pin.IN, pull=Pin.PULL_UP)
16    int_pin.irq(handler=imu_int_handler, trigger=Pin.IRQ_RISING)
17
18# Vibration detection example
19UCF_FILE = "lsm6dsox_vibration_monitoring.ucf"
20UCF_LABELS = {0: "no vibration", 1: "low vibration", 2: "high vibration"}
21# NOTE: Selected data rate and scale must match the MLC data rate and scale.
22lsm = LSM6DSOX(
23    SPI(5),
24    cs=Pin("PF6", Pin.OUT_PP, Pin.PULL_UP),
25    gyro_odr=26,
26    accel_odr=26,
27    gyro_scale=2000,
28    accel_scale=4,
29    ucf=UCF_FILE,
30)
31
32print("MLC configured...")
33
34while True:
35    if INT_MODE:
36        if INT_FLAG:
37            INT_FLAG = False
38            print(UCF_LABELS[lsm.mlc_output()[0]])
39    else:
40        buf = lsm.mlc_output()
41        if buf is not None:
42            print(UCF_LABELS[buf[0]])

In the OpenMV IDE Serial Monitor, the inference results will be printed after a vibration event.

You can download and test many other pre-trained models available in this repository.

With Arduino IDE

First, to use this sensor with the Arduino IDE, you need to install the

Arduino_LSM6DSOX

LSM6DSOX

The example code below shows how to get acceleration and angular velocity data from the onboard IMU and stream it to the IDE's Serial Monitor and Serial Plotter.

Copy
1#include <Arduino_LSM6DSOX.h>
2
3void setup() {
4  Serial.begin(9600);
5  while (!Serial)
6    ;
7
8  if (!IMU.begin()) {
9    Serial.println("Failed to initialize IMU!");
10
11    while (1)
12      ;
13  }
14
15  Serial.print("Accelerometer sample rate = ");
16  Serial.print(IMU.accelerationSampleRate());
17  Serial.println(" Hz");
18  Serial.println();
19  Serial.println("Acceleration in g's");
20  Serial.println("X\tY\tZ");
21
22  Serial.print("Gyroscope sample rate = ");
23  Serial.print(IMU.gyroscopeSampleRate());
24  Serial.println(" Hz");
25  Serial.println();
26  Serial.println("Gyroscope in degrees/second");
27  Serial.println("X\tY\tZ");
28
29  delay(3000);  // Wait 3 seconds
30}
31
32void loop() {
33  float a_x, a_y, a_z;
34
35  if (IMU.accelerationAvailable()) {
36    IMU.readAcceleration(a_x, a_y, a_z);
37
38    Serial.print("acc_X:");
39    Serial.print(a_x);
40    Serial.print(",");
41    Serial.print("acc_Y:");
42    Serial.print(a_y);
43    Serial.print(",");
44    Serial.print("acc_Z:");
45    Serial.println(a_z);
46  }
47  float g_x, g_y, g_z;
48
49  if (IMU.gyroscopeAvailable()) {
50    IMU.readGyroscope(g_x, g_y, g_z);
51
52    Serial.print("gyro_X:");
53    Serial.print(g_x);
54    Serial.print(",");
55    Serial.print("gyro_Y:");
56    Serial.print(g_y);
57    Serial.print(",");
58    Serial.print("gyro_Z:");
59    Serial.println(g_z);
60  }
61}

To test a Machine Learning model on the Arduino IDE, navigate to File > Examples > MLC > NiclaVision_MLC_Motion_Intesity and it will identify three scenarios:

Stationary

Medium Intensity

High Intensity

Microphone

The onboard high-performance microphone of the Nicla Vision is the MP34DT06JTR from ST®. It is specifically designed for applications that require high-quality audio recording and accurate voice detection, such as voice-controlled Internet of Things (IoT) devices, smart home systems, and mobile devices.

Using OpenMV

The OpenMV IDE includes some examples to get started using the Nicla Vision onboard microphone that can be found on File > Examples > Audio. We are going to use the one called

micro_speech.py

First, download the pre-trained model file from the example repository and copy it to the Nicla Vision storage drive.

Reset the board and run the following code on the OpenMV IDE.

Copy
1import audio
2import time
3import tf
4import micro_speech
5import pyb
6
7labels = ["Silence", "Unknown", "Yes", "No"]
8
9led_red = pyb.LED(1)
10led_green = pyb.LED(2)
11
12model = tf.load("/model.tflite")
13speech = micro_speech.MicroSpeech()
14audio.init(channels=1, frequency=16000, gain=24, highpass=0.9883)
15
16# Start audio streaming
17audio.start_streaming(speech.audio_callback)
18
19while True:
20    # Run micro-speech without a timeout and filter detections by label index.
21    idx = speech.listen(model, timeout=0, threshold=0.70, filter=[2, 3])
22    led = led_green if idx == 2 else led_red
23    print(labels[idx])
24    for i in range(0, 4):
25        led.on()
26        time.sleep_ms(25)
27        led.off()
28        time.sleep_ms(25)
29
30# Stop streaming
31audio.stop_streaming()

After running the code, the matches will be printed on the Serial Monitor if the board hears a

No

Yes

Using Arduino IDE

The Arduino IDE includes a simple example to visualize raw data from the PDM microphone. To test it, navigate to File > Examples > PDM > PDMSerialPlotter.

Copy
1#include <PDM.h>
2
3// default number of output channels
4static const char channels = 1;
5
6// default PCM output frequency
7static const int frequency = 16000;
8
9// Buffer to read samples into, each sample is 16-bits
10short sampleBuffer[512];
11
12// Number of audio samples read
13volatile int samplesRead;
14
15void setup() {
16  Serial.begin(9600);
17  while (!Serial);
18
19  // Configure the data receive callback
20  PDM.onReceive(onPDMdata);
21
22  // Optionally set the gain
23  // Defaults to 20 on the BLE Sense and 24 on the Portenta Vision Shield
24  // PDM.setGain(30);
25
26  // Initialize PDM with:
27  // - one channel (mono mode)
28  // - a 16 kHz sample rate for the Arduino Nano 33 BLE Sense
29  // - a 32 kHz or 64 kHz sample rate for the Arduino Portenta Vision Shield
30  if (!PDM.begin(channels, frequency)) {
31    Serial.println("Failed to start PDM!");
32    while (1);
33  }
34}
35
36void loop() {
37  // Wait for samples to be read
38  if (samplesRead) {
39
40    // Print samples to the serial monitor or plotter
41    for (int i = 0; i < samplesRead; i++) {
42      if(channels == 2) {
43        Serial.print("L:");
44        Serial.print(sampleBuffer[i]);
45        Serial.print(" R:");
46        i++;
47      }
48      Serial.println(sampleBuffer[i]);
49    }
50
51    // Clear the read count
52    samplesRead = 0;
53  }
54}
55
56/**
57 * Callback function to process the data from the PDM microphone.
58 * NOTE: This callback is executed as part of an ISR.
59 * Therefore using `Serial` to print messages inside this function isn't supported.
60 * */
61void onPDMdata() {
62  // Query the number of available bytes
63  int bytesAvailable = PDM.available();
64
65  // Read into the sample buffer
66  PDM.read(sampleBuffer, bytesAvailable);
67
68  // 16-bit, 2 bytes per sample
69  samplesRead = bytesAvailable / 2;
70}

Upload the example code to the Nicla Vision and open the Serial Plotter to see the sound wave output.

Time of Flight (Distance) Sensor

The onboard ToF sensor of the Nicla Vision is the VL53L1CBV0FY from ST®. It adds accurate and low power ranging capabilities to the Nicla Vision. The invisible near-infrared VCSEL laser (including the analog driver) is encapsulated with receiving optics in an all-in-one small module located below the camera.

Here are listed the sensor's main features:

• Up to 400 cm distance measurement
• Up to 50 Hz ranging frequency
• 27° field-of-view (FoV)

With OpenMV

The OpenMV IDE includes an example to start using the ToF sensor. To test it, navigate to File > Examples > Sensors > vl53l1x_tof and run it on the Nicla Vision.

Copy
1from machine import I2C
2from vl53l1x import VL53L1X
3import time
4
5tof = VL53L1X(I2C(2))
6
7while True:
8    print(f"Distance: {tof.read()}mm")
9    time.sleep_ms(50)

With Arduino IDE

To use the ToF sensor with the Arduino IDE, install the

VL53L1X

Once installed, you will be able to compile and upload the example code below to your Nicla Vision.

The distance measured by the sensor will be printed on the IDE's Serial Monitor, and the built-in LED will blink proportionally to that distance.

Copy
1#include "VL53L1X.h"
2VL53L1X proximity;
3
4bool blinkState = false;
5int reading = 0;
6int timeStart = 0;
7int blinkTime = 2000;
8
9void setup() {
10  Serial.begin(115200);
11  Wire1.begin();
12  Wire1.setClock(400000); // use 400 kHz I2C
13  proximity.setBus(&Wire1);
14
15
16  pinMode(LEDB, OUTPUT);
17  digitalWrite(LEDB, blinkState);
18
19  if (!proximity.init()) {
20    Serial.println("Failed to detect and initialize sensor!");
21    while (1);
22  }
23
24  proximity.setDistanceMode(VL53L1X::Long);
25  proximity.setMeasurementTimingBudget(50000);
26  proximity.startContinuous(50);
27}
28
29void loop() {
30  reading = proximity.read();
31  Serial.print(reading);
32  Serial.println(" mm");
33
34  if (millis() - timeStart >= reading) {
35    digitalWrite(LEDB, blinkState);
36    timeStart = millis();
37
38    blinkState = !blinkState;
39  }
40}

Camera

The Nicla Vision's main feature is its onboard 2MP camera, based on the GC2145 color rolling shutter image sensor. It is perfect for Machine Learning applications such as object detection, image classification, machine/computer vision, robotics, IoT, and more.

The Nicla Vision is primarily intended to be used with the OpenMV MicroPython ecosystem. So, it's recommended to use this IDE for machine vision applications.

With OpenMV

The OpenMV IDE is designed to work specifically with machine/computer vision hardware, it is optimized for easy and fast development of image processing applications with a MicroPython framework and streaming monitors, color data graphics, and more.

The Nicla Vision uses a 2MP camera sensor, meaning its maximum resolution is 1920x1080 pixels. However, the effective resolution is 1616(H) × 1232(V).

Here we have the minimum code necessary to make the camera work streaming live video on the OpenMV IDE:

Copy
1import sensor
2import time
3
4sensor.reset()  # Reset and initialize the sensor.
5sensor.set_pixformat(sensor.RGB565)  # Set pixel format to RGB565 (or GRAYSCALE)
6sensor.set_framesize(sensor.QVGA)  # Set frame size to QVGA (320x240)
7sensor.skip_frames(time=2000)  # Wait for settings take effect.
8clock = time.clock()  # Create a clock object to track the FPS.
9
10while True:
11    clock.tick()  # Update the FPS clock.
12    img = sensor.snapshot()  # Take a picture and return the image.
13    print(clock.fps())  # Note: OpenMV Cam runs about half as fast when connected
14    # to the IDE. The FPS should increase once disconnected.

From the above example script, we can highlight the main functions:

• sensor.set_pixformat(<Sensor>)

  lets you set the pixel format for the camera sensor. The Nicla Vision is compatible with these:

  sensor.GRAYSCALE

  ,

  sensor.RGB565

  ,

  sensor.BAYER

  , and

  sensor.YUV422

  .

  To define the pixel format to any of the supported ones, just add it to the

  set_pixformat

  function argument.

• sensor.set_framesize(<Resolution>)

  lets you define the image frame size in terms of pixels. Here you can find all the different options.

• sensor.snapshot()

  lets you take a picture and return the image so you can save it, stream it or process it.

The example code below lets you take a picture and save it on the Nicla Vision local storage as

example.jpg

Copy
1import sensor
2import time
3import machine
4
5sensor.reset()  # Reset and initialize the sensor.
6sensor.set_pixformat(sensor.RGB565)  # Set pixel format to RGB565 (or GRAYSCALE)
7sensor.set_framesize(sensor.QVGA)  # Set frame size to QVGA (320x240)
8sensor.skip_frames(time=2000)  # Wait for settings take effect.
9
10led = machine.LED("LED_BLUE")
11
12start = time.ticks_ms()
13while time.ticks_diff(time.ticks_ms(), start) < 3000:
14    sensor.snapshot()
15    led.toggle()
16
17led.off()
18
19img = sensor.snapshot()
20img.save("example.jpg")  # or "example.bmp" (or others)
21
22raise (Exception("Please reset the camera to see the new file."))

After the snapshot is taken, reset the board by pressing the reset button and the image will be on the board storage drive.

The example code below lets you record a video and save it on the Nicla Vision local storage as

example.mjpeg

Copy
1import sensor
2import time
3import mjpeg
4import machine
5
6sensor.reset()  # Reset and initialize the sensor.
7sensor.set_pixformat(sensor.RGB565)  # Set pixel format to RGB565 (or GRAYSCALE)
8sensor.set_framesize(sensor.QVGA)  # Set frame size to QVGA (320x240)
9sensor.skip_frames(time=2000)  # Wait for settings take effect.
10
11led = machine.LED("LED_RED")
12
13led.on()
14m = mjpeg.Mjpeg("example.mjpeg")
15
16clock = time.clock()  # Create a clock object to track the FPS.
17for i in range(200):
18    clock.tick()
19    m.add_frame(sensor.snapshot())
20    print(clock.fps())
21
22m.close(clock.fps())
23led.off()
24
25raise (Exception("Please reset the camera to see the new file."))

After the video is recorded, reset the board by pressing the reset button and the file will be on the board storage drive.

We recommend using VLC to play the video due to the format.

The next example lets you live stream what the camera sees through HTTP so you can watch it on your favorite browser from any device connected to the same network as the Nicla Vision.

Make sure to fill in the

SSID

KEY

Copy
1import sensor
2import time
3import network
4import socket
5
6SSID = "*********"  # Network SSID
7KEY = "************"  # Network key
8HOST = ""  # Use first available interface
9PORT = 8080  # Arbitrary non-privileged port
10
11# Init sensor
12sensor.reset()
13sensor.set_framesize(sensor.QVGA)
14sensor.set_pixformat(sensor.RGB565)
15
16# Init wlan module and connect to network
17wlan = network.WLAN(network.STA_IF)
18wlan.active(True)
19wlan.connect(SSID, KEY)
20
21while not wlan.isconnected():
22    print('Trying to connect to "{:s}"...'.format(SSID))
23    time.sleep_ms(1000)
24
25# We should have a valid IP now via DHCP
26print("WiFi Connected ", wlan.ifconfig())
27
28# Create server socket
29s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
30s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, True)
31
32# Bind and listen
33s.bind([HOST, PORT])
34s.listen(5)
35
36# Set server socket to blocking
37s.setblocking(True)
38
39
40def start_streaming(s):
41    print("Waiting for connections..")
42    client, addr = s.accept()
43    # set client socket timeout to 5s
44    client.settimeout(5.0)
45    print("Connected to " + addr[0] + ":" + str(addr[1]))
46
47    # Read request from client
48    data = client.recv(1024)
49    # Should parse client request here
50
51    # Send multipart header
52    client.sendall(
53        "HTTP/1.1 200 OK\r\n"
54        "Server: OpenMV\r\n"
55        "Content-Type: multipart/x-mixed-replace;boundary=openmv\r\n"
56        "Cache-Control: no-cache\r\n"
57        "Pragma: no-cache\r\n\r\n"
58    )
59
60    # FPS clock
61    clock = time.clock()
62
63    # Start streaming images
64    # NOTE: Disable IDE preview to increase streaming FPS.
65    while True:
66        clock.tick()  # Track elapsed milliseconds between snapshots().
67        frame = sensor.snapshot()
68        cframe = frame.compressed(quality=35)
69        header = (
70            "\r\n--openmv\r\n"
71            "Content-Type: image/jpeg\r\n"
72            "Content-Length:" + str(cframe.size()) + "\r\n\r\n"
73        )
74        client.sendall(header)
75        client.sendall(cframe)
76        print(clock.fps())
77
78
79while True:
80    try:
81        start_streaming(s)
82    except OSError as e:
83        print("socket error: ", e)
84        # sys.print_exception(e)

Once you run this script, the Nicla Vision IP address will be printed on the OpenMV serial monitor after the Wi-Fi® connection processes.

To watch the live stream, enter the device IP address followed by the

:8080

<Nicla Vision IP>:8080

To expand your knowledge using the Nicla Vision camera with MicroPython, try other built-in examples within the OpenMV IDE.

Machine Learning Tools

The Nicla Vision is a ready-to-use, standalone camera board, ready for analyzing and processing images on the Edge. Thanks to its 2MP color camera, smart 6-axis motion sensor, integrated microphone, and distance sensor, it is suitable for almost infinite machine-learning applications.

Creating this type of application has never been easier thanks to our Machine Learning Tool powered by Edge Impulse®, where we can easily create in a No-Code environment, Audio, Motion, Proximity and Image processing models.

The first step to start creating awesome artificial intelligence and machine learning projects is to create an Arduino Cloud account.

There you will find a dedicated integration called Machine Learning Tools.

Once in, create a new project and give it a name.

Enter your newly created project and the landing page will look like the following:

Edge Impulse® Environment Setup

Now, it is time to set up the Edge Impulse® environment on your PC. For this, follow these instructions to install the Edge Impulse CLI.

For Windows users: make sure to install Visual Studio Community andVisual Studio Build Tools.

• Download and install the latest Arduino CLI from here. (Video Guide for Windows)

• Download the latest Edge Impulse® firmware, and unzip the file.

• Open the flash script for your operating system (

  flash_windows.bat

  ,

  flash_mac.command

  or

  flash_linux.sh

  ) to flash the firmware.

• To test if the Edge Impulse CLI was installed correctly, open the Command Prompt or your favorite terminal and run:

  edge-impulse-daemon

  If everything went okay, you should be asked for your account credentials.

  

• Enter your account username or e-mail address and your password.

• Select the project you have created on the Arduino ML Tools, it will be listed.

• Give your device a name and wait for it to connect to the platform.

Uploading Sensor Data

The first thing to start developing a machine learning project is to create a dataset for your model. This means, uploading data to your model from any of the Nicla Vision sensors.

To upload data from your Nicla Vision on the Machine Learning Tools platform, navigate to Data Acquisition.

In this section, you will be able to select the Nicla Vision onboard sensors individually or several interesting combinations.

This is the supported sensors list:

• Built-in microphone
• Inertial (IMU)
• ADC sensor (A0)
• Proximity sensor
• Camera (different resolutions)

Now you know how to start with our Machine Learning Tools creating your dataset from scratch, you can get inspired by some of our ML projects listed below:

• Image Classification with Edge Impulse® (Article).
• Glass-Breaking Detector using Edge Impulse® (Video).

Communication

This section of the user manual covers the different communication protocols that are supported by the Nicla Vision, including the Serial Peripheral Interface (SPI), Inter-Integrated Circuit (I2C), Universal Asynchronous Receiver-Transmitter (UART), and Bluetooth® Low Energy; communication via the onboard ESLOV connector is also explained in this section. The Nicla Vision features dedicated pins for each communication protocol, making connecting and communicating with different components, peripherals, and sensors easy.

SPI

The Nicla Vision supports SPI communication, which allows data transmission between the board and other SPI-compatible devices. The pins used in the Nicla Vision for the SPI communication protocol are the following:

Please, refer to the board pinout section of the user manual to localize them on the board.

With OpenMV

Import the

SPI

pyb

time

Pin

Copy
1import time
2from pyb import Pin, SPI

Before your infinite loop, configure the chip select (

CS

Master

Copy
1spi = SPI(4, SPI.MASTER, baudrate=int(480000000 / 256), polarity=0, phase=0)

To send data over SPI, use:

Copy
1spi.send(<data>)  # add the data (integer or buffer) to be sent as the argument

To receive data over SPI, use:

Copy
1spi.recv(<bytes>) # add the number of bytes to be received as the argument

Here is a simple example showing how to send data over SPI.

Copy
1import time
2from pyb import Pin, SPI
3
4cs = Pin("SS", Pin.OUT_OD) # CS pin = PE11
5
6spi = SPI(4, SPI.MASTER, baudrate=int(480000000 / 256), polarity=0, phase=0)
7
8while True:
9    # Replace with the target device's address
10    address = 0x35
11    # Replace with the value to send
12    value = 0xFA
13    # Pull the CS pin LOW to select the device
14    cs.low()
15    # Send the address
16    spi.send(address)
17    # Send the value
18    spi.send(value)
19    # Pull the CS pin HIGH to unselect the device
20    cs.high()
21    
22    time.sleep_ms(1000)

The example code above should output this:

With Arduino IDE

Include the

SPI

Copy
1#include <SPI.h>

In the

setup()

CS

Copy
1void setup() {
2  // Set the chip select pin as output
3  pinMode(SS, OUTPUT); 
4  // Pull the CS pin HIGH to unselect the device
5  digitalWrite(SS, HIGH); 
6  
7  // Initialize the SPI communication
8  SPI.begin();
9}

To transmit data to an SPI-compatible device, you can use the following commands:

Copy
1// Replace with the target device's address
2byte address = 0x35; 
3// Replace with the value to send
4byte value = 0xFA; 
5// Pull the CS pin LOW to select the device
6digitalWrite(SS, LOW); 
7// Send the address
8SPI.transfer(address); 
9// Send the value
10SPI.transfer(value); 
11// Pull the CS pin HIGH to unselect the device
12digitalWrite(SS, HIGH);

Here is the complete sketch for a simple SPI communication:

Copy
1#include <SPI.h>
2
3void setup(){
4  // Set the chip select pin as output
5  pinMode(SS, OUTPUT); 
6  // Pull the CS pin HIGH to unselect the device
7  digitalWrite(SS, HIGH); 
8  
9  // Initialize the SPI communication
10  SPI.begin();
11}
12
13void loop(){
14  // Replace with the target device's address
15  byte address = 0x35; 
16  // Replace with the value to send
17  byte value = 0xFA; 
18  // Pull the CS pin LOW to select the device
19  digitalWrite(SS, LOW); 
20  // Send the address
21  SPI.transfer(address); 
22  // Send the value
23  SPI.transfer(value); 
24  // Pull the CS pin HIGH to unselect the device
25  digitalWrite(SS, HIGH); 
26  delay(1000);
27}

The example code above should output this:

I2C

The Nicla Vision supports I2C communication, which allows data transmission between the board and other I2C-compatible devices. The pins used in the Nicla Vision for the I2C communication protocol are the following:

Please, refer to the board pinout section of the user manual to localize them on the board. The I2C pins are also available through the onboard ESLOV connector of the Nicla Vision.

With OpenMV

To use I2C communication with OpenMV, import

I2C

pyb

Copy
1from pyb import I2C

Create the I2C objects and initialize them attached to a specific bus, below are some of the available commands to do it.

Copy
1i2c = I2C(1)                         # create on bus 1
2i2c = I2C(1, I2C.MASTER)             # create and init as a master
3i2c.init(I2C.MASTER, baudrate=20000) # init as a master
4i2c.init(I2C.SLAVE, addr=0x42)       # init as a slave with given address
5i2c.deinit()                         # turn off the peripheral

The basic methods are

send

recv

Copy
1# For Masters / Controllers (must include addr in send)
2i2c.send('abc', 0x42)      # send 3 bytes to device on address 0x42
3
4# For Slaves / Peripherals
5i2c.send('abc')      # send 3 bytes
6i2c.send(0x42)       # send a single byte, given by the number
7
8data = i2c.recv(3)   # receive 3 bytes

This is a simple example showing how to send data over I2C from a master to a slave.

Copy
1from pyb import I2C
2
3i2c = I2C(1, I2C.MASTER)
4
5buf = bytearray(2)
6
7buf[0] = 0x00
8buf[1] = 0xFA
9
10i2c.send(buf, 0x35)