CSC477 / CSC2630 Introduction to Mobile Robotics

Week #1: Introduction, Sensors & Actuators

Florian Shkurti

Today’s agenda

  • Introduction
  • Administrivia
    • Office hours
    • Tutorials
    • Assignment descriptions
    • Prerequisites
  • Topics covered by the course
  • Sensors and Actuators
  • Quiz about background and interests

Your TAs

Yewon Lee
MSc student
Computer Science, UofT

Yasasa Abeysirigoonawardena
MSc student
Computer Science, UofT

Radian Gondokaryono
PhD student
Computer Science, UofT

My lab: Robot Vision and Learning (RVL)

Mission: create algorithms that enable robots to learn to act intelligently in outdoor environments and alongside humans

How I became interested in robotics

Mars Exploration Rover

 

RoboCup, small-sized league

How I became interested in robotics

How I became interested in robotics

Today you have

Factory Automation

Autonomous warehouse robots at Amazon

Autonomous arms at Tesla

Pipe Inspection

Manually-controlled inspection robots

Nuclear Disaster Cleanup

Remote-controlled cleaning robot at Fukushima Daiichi, 2011

Nuclear Disaster Cleanup

Remote-controlled cleaning robot at Fukushima Daiichi, 2011

Nuclear Disaster Cleanup

Remote-controlled cleaning robot at Chernobyl, 1986

Aerial Package Delivery

Aerial First-Aid Delivery

Smart Wheelchairs

Robot Surgery

daVinci robot-assisted surgery

Precision Agriculture

farmbot.io

Self-driving Trucks

Mining Operations

Oil Spill Containment

BP Deepwater Horizon Spill, Gulf of Mexico, 2010

Autonomy vs. Remote Control

  • Q: When is full or partial autonomy necessary?

  • Q: When is remote control preferred?

Today’s agenda

  • Introduction
  • Administrivia
    • Office hours
    • Tutorials
    • Assignment descriptions
    • Prerequisites
  • Topics covered by the course
  • Sensors and Actuators
  • Quiz about background and interests

Prerequisites

  • Software Engineering
    • Loops, conditionals, classes, modularity
    • Lists, hash maps/dictionaries, trees
    • Threads, callbacks, remote procedure calls, serialization
  • Linear Algebra
    • Matrix multiplication and inversion, determinant
    • Solving systems of equations, Gaussian elimination
    • Matrix decompositions: Cholesky, QR
    • Least squares
  • Basic Probability Theory
    • Multivariate distributions, especially Gaussians
    • Conditional probability, Bayes’ rule
    • Maximum likelihood estimation

Prerequisites

Currently

Required: CSC209H5; STA256H5; MAT223H5/MAT240H5; MAT232H5; CSC376

Recommended: MAT224H5; CSC384H5; CSC311H5;

4 Assignments

  • ~80% coding and the rest theory

  • Starter code will be provided

  • Bonus questions will be provided

  • Accepted languages: Python, C++

  • You’re going to learn ROS (Robot Operating System) and use the Gazebo simulator

  • You’re also going to learn numpy and scipy

  • About 2 weeks to work on each

ROS + Gazebo simulation

7 Quizzes

  • 5-10 mins to complete them

  • Not cumulative in terms of material. They cover only one lecture

  • Meant to check whether you have understood basic concepts

Evaluation

CSC477

  • 4 assignments, 15% each = 60%

  • 7 quizzes, 2% each = 14%

  • 1 final exam = 26%


CSC2630

  • 3 assignments, 15% each = 45%

  • 7 quizzes, 2% each = 14%

  • 1 final project = 41%

Office Hours (Zoom)

  • Florian: Thursdays 3-4pm
  • Yewon: Tuesdays 11-12pm
  • Yasasa: Fridays 11-12pm

  • Office hours will begin next week

Online communication

Today’s agenda

  • Introduction
  • Administrivia
    • Office hours
    • Tutorials
    • Assignment descriptions
    • Prerequisites
  • Topics covered by the course
  • Sensors and Actuators
  • Quiz about background and interests

Main topics to be covered

Covered \(\qquad \qquad \qquad\) Not Covered

Main topics to be covered

Main question: what is the next state given the curent state and controls?

Main topics to be covered

Main question: what are the controls that will take the system from state A to B?

Main topics to be covered

Main topics to be covered

  • Known: robot’s position and orientation
  • Want to estimate: a map of the environment from laser measurements

  • Occupancy grid mapping

Main topics to be covered

orb slam video

https://www.youtube.com/watch?v=_9VcvGybsDA

Lecture topics \(\qquad \qquad\) Tutorials

Intro to the Robot Operating System (ROS)

Refresher on linear algebra and least squares

Refresher on basic probability and continuous distributions

How to align 3D pointclouds. Demo of the PCL library

How to implement a Kalman Filter

How to implement a Particle Filter

How to approximate functions

Assignments


A1: Designing a feedback controller for wall-following

Assignments



A1: Designing a feedback controller for wall-following


A2: Implementing path-planning and feedback control algorithms

Assignments


 


A1: Designing a feedback controller for wall-following


A2: Implementing path-planning and feedback control algorithms


A3: Occupancy grid mapping with known robot location

A4: Localization in a known map using particle filters

Today’s agenda

  • Introduction
  • Administrivia
    • Office hours
    • Tutorials
    • Assignment descriptions
    • Prerequisites
  • Topics covered by the course
  • Sensors and Actuators
  • Quiz about background and interests

Sensors and Actuators

  • Sensors:
    • Characteristics and types
    • Measurement noise
    • Required bandwidth

  • Actuators:
    • Types of motors
    • Pulse-Width Modulation

Sensors

  • Devices that can sense and measure physical properties of the environment.

  • Key phenomenon is transduction (conversion of energy from one form to another). E.g.:

    • Imaging sensors: light to pixel voltages
    • Depth sensors: mechanical pressure to voltage

  • Measurements are noisy, and difficult to interpret

Sensors: general characteristics

  • Sensitivity: (change of output) ÷ (change of input)
  • Linearity: constancy of (output ÷ input)
  • Measurement range: [min, max] or {min, max}
  • Response time: time required for input change to cause output change
  • Accuracy: difference between measurement and actual
  • Repeatability/Drift: difference between repeated measures
  • Resolution: smallest observable increment
  • Bandwidth: required rate of data transfer
  • SNR: signal-to-noise ratio

Sensors: vision

CCD image sensor

CCD (charge-coupled device) imaging sensors:

  • Capacitor array accumulates electric charge proportional to light intensity.
  • Each capacitor’s charge is transferred to its neighbor.
  • Last capacitor’s charge gets amplified and output as voltage.
  • (+) High-quality, low-noise images
  • (-) Higher power consumption
  • (-) Slow readout
  • (-) Specialized fabrication

voltage → analog-to-digital converter → pixel value in {0, 255}

CMOS (complementary metal-oxide semi-conductor) imaging sensors:

  • One amplifier per pixel
  • (+) Low power
  • (+) Fast readout
  • (+) Easier to fabricate
  • (-) Poor low-light sensitivity
  • (-) Higher noise

Global vs. Rolling Shutter

Shutter = mechanism that allows light to hit the imaging sensor

Shutter “speed” = Exposure time = time duration in which the sensor is exposed to light

Rolling shutter

Reading RGB images from a camera

Each pixel contains an intensity value from 0…255

600 x 1000 pixels

600 x 1000 pixels

600 x 1000 pixels

Reading RGB images from a camera

Each pixel contains an intensity value from 0…255

\(\to\) A matrix of 600 x 1000 x 3 = ~ 1.8 million numbers

600 x 1000 pixels

600 x 1000 pixels

600 x 1000 pixels

Computer/robot vision

Structured numbers

  1. I’m seeing a parrot

  2. I’m seeing a toy bicycle

  3. The parrot is riding the bicycle

  4. The bicycle is on top of a desk

  5. Is this physically plausible?

  6. Where is the parrot in 3D w.r.t. the camera?

  7. Where will the parrot go next?

  8. What is the speed of the parrot?

Conclusions/Inference/Deduction/Estimation

Camera lenses

  • Lens determines:
    • image distortion
    • focus
    • sharpness or blur
  • Lens characteristics:
    • focal length
    • aperture
    • depth-of-field

Pinhole Camera Model

We know approximately how a 3D point (X,Y,Z) projects to pixel (x,y)
We call this the pinhole projection model

(1) Perspective projection [x,y] = 𝜋(X,Y,Z)

http://www.cim.mcgill.ca/%7Elanger/558.html

By similar triangles: x/f = X/Z

So, x = f * X/Z and similarly y = f * Y/Z

Problem: we just lost depth (Z) information by doing this projection, i.e. depth is now uncertain.

(2) Lens distortion

(2) Estimating parameters of lens distortion

Non-pinhole cameras: thin lens model

Unlike the pinhole camera, this is able to model blur.

http://www.cim.mcgill.ca/%7Elanger/558.html

Beyond the visible spectrum: infrared cameras

Beyond the visible spectrum: infrared cameras

Drawback: Doesn’t work underwater

Beyond the visible spectrum: infrared cameras

Beyond the visible spectrum: RGBD cameras

Main ideas:

  • Active sensing
  • Projector emits infrared light in the scene
  • Infrared sensor reads the infrared light
  • Deformation of the expected pattern allows computation of the depth

Beyond the visible spectrum: RGBD cameras

Drawbacks:

  • Does not work well outdoors, sunlight saturates its measurements
  • Maximum range is [0.5, 8] meters

Advantages:

  • Real-time depth estimation at 30Hz
  • Cheap

Beyond the visible spectrum: RGBD cameras

Enabled a wave of research, applications, and video games, based on real-time skeleton tracking

Beyond the visible spectrum: RGBD cameras

Despite their drawbacks RGBD sensors have been extensively used in robotics.

3D LIDAR (Light detection and ranging)

Produces a pointcloud of 3D points and intensities

  • (x,y,z) in the laser’s frame of reference
  • Intensity is related to the material of the object that reflects the light

Works based on time-of-flight for each beam to return back to the scanner

Not very robust to adverse weather conditions: rain, snow, smoke, fog etc.

Used in most self-driving cars today for obstacle detection. Range < 100m.

Usually around 1million points in a single pointcloud

2D LIDAR (Light detection and ranging)

Produces a scan of 2D points and intensities

  • (x,y) in the laser’s frame of reference
  • Intensity is related to the material of the object that reflects the light

Certain surfaces are problematic for LIDAR: e.g. glass

Lots of moving parts: motors quickly rotate the laser beam and once complete (angle bound reached) a scan is returned. I.e. points are not strictly speaking time-synchronized, even though we usually treat them as such.

Usually around 1024 points in a single scan.

Inertial Sensors

  • Gyroscopes, Accelerometers, Magnetometers

  • Inertial Measurement Unit (IMU)

  • Perhaps the most important sensor for 3D navigation, along with the GPS

  • Without IMUs, plane autopilots would be much harder, if not impossible, to build

Gyroscopes

  • Measure angular velocity in the body frame

  • Often affected by noise and bias

\[ w_\text{measured}(t) = w_\text{true}(t) + b_g(t) + n_g(t) \]

  • We integrate it to get 3D orientation (Euler angles, quaternions rotation matrices), but there is drift due to noise and bias

Accelerometers

  • Measure linear acceleration relative to freefall (measured in g)

  • A free-falling accelerometer in a vacuum would measure zero g

  • An accelerometer resting on the surface of the earth would measure 1g

  • Also affected by bias and noise.

  • Double integration to get position is very noisy. Errors grow quadratically with time.

Magnetometers

Drawbacks:

  • Needs careful calibration
  • Needs to be placed away from moving metal parts, motors

Advantages:

  • Can be used as a compass for absolute heading

Inertial Measurement Unit

  • Combines measurements from accelerometer, gyroscope, and magnetometer to output an estimate of orientation with reduced drift.

  • Does not typically provide a position estimate, due to double integration.

  • Runs at 100-1000Hz

  • Expect yaw drift of 5-10 deg/hour on most modern low-end IMUs

Global Positioning System: Satellites

  • Each GPS satellite periodically transmits:

  • [Coarse/Acquisition code] A 1023-bit pseudorandom binary sequence (PRN code), which repeats every 1 ms, unique for each satellite (no correlation with other satellites).

  • [Navigation frame] A 1500-bit packet that contains

    • GPS date, time, satellite health
    • Detailed orbital data for the satellite, accurate for the next ~4hrs
    • PRN codes and status of all satellites in the network
    • Takes 12.5mins to transmit

  • [Precision code] A 6.2-terabit code for military use.

  • Carrier frequencies are 1575.42 MHz (L1) and 1227.60 MHz (L2)

Global Positioning System: Receivers

  • Each (civilian) GPS receiver:
    • Knows the PRN codes for each satellite in advance
    • Correlates received PRN signal with database PRN signal → time shift → noisy distance to satellite
    • If 4 or more satellite PRN codes are received, it does trilateration to compute latitude and longitude

Global Positioning System: Receivers and Dilution of Precision

Hall Effect Sensor

  • Varies its voltage in response to a magnetic field
  • Used as a proximity switch, to measure a full rotation of a wheel for example
  • Used to measure rate of rotation of wheels

Rotary Encoder

  • Contains an analog to digital converter for encoding the angle of a shaft/motor/axle

  • Usually outputs the discretized absolute angle of the shaft/motor/axle

  • Useful in order to know where different shafts are relative to each to each other.

Example: flippers on the Aqua robot

Actuators

DC (direct current) motor

Servo motor

Stepper motor

They turn continuously at high RPM (revolutions per minute) when voltage is applied. Used in quadrotors and planes, model cars etc.

Usually includes: DC motor, gears, control circuit, position feedback
Precise control without free rotation (e.g. robot arms, boat rudders) Limited turning range: 180 degrees

Positioning feedback and no positioning errors.
Rotates by a predefined step angle.
Requires external control circuit.
Precise control without free rotation.
Constant holding torque without powering the motor (good for robot arms or weight-carrying systems).

Pulse Width Modulation

Used for creating analog/continuous behavior when voltage applied is discrete.
Main idea: turn on and off the motor fast enough so average voltage is the desired target.
Used in dimming LEDs, controlling the speed of DC motors, controlling the position of servo motors.

Today’s agenda

  • Introduction
  • Administrivia
    • Office hours
    • Tutorials
    • Assignment descriptions
    • Prerequisites
  • Topics covered by the course
  • Sensors and Actuators
  • Quiz about background and interests