Trouble for nothing

I bought one of these cameras from a famous Chinese e-commerce site installing it on the roof of my house. This camera is produced by Huisun and it’s a clone of a famous model produced by Dahua. It has a WEB interface (unfortunately usable only in conjunction with an ActiveX object) and permits to forward an HD video stream on the local network using a wired connection. After one year of good and loyal service I decided, during a maintenance stop, to investigate more on its internal structure and obtain, if possible, a shell access on it.

A scan with Nmap reveals a lot of opened ports with some interesting daemons listening on them:
(07/04/2019 note: I wrote this article one year ago and I’m not able to find anymore the original screenshots of the terminal, so I report here the textual output of the commands extracted from collected outputs.)

PORT         STATE   SERVICE           VERSION
80/tcp       open    http              tinyproxy 1.8.3
81/tcp       open    hosts2-ns         Boa HTTPd 0.94.13
443/tcp      open    https
554/tcp      open    rtsp
3001/tcp     open    nessus
3232/tcp     open    mdtp              Busybox telnetd
8000/tcp     open    http-alt
9090/tcp     open    zeus-admin        gSOAP 2.8
9800/tcp     open    davsrc
9897/tcp     open    unknown
27777/tcp    open    unknown
34567/tcp    open    dhanalakshmi
37777/tcp    open    unknown

The camera replies to HTTP requests on port 80, 81, 443, and 9897. On 81/TCP (and 443/TCP) it’s listening an HTTP Boa server1: it’s a very simple HTTP server not actively developed since 2005.
The other HTTP daemon, the web-proxy Tinyproxy2, is listening on 80/TCP: probably (spoiler: it’s the correct hypothesis as I discovered later) it is used in order to redirect different requested URLs to the different services listening on the camera.
Ports 554/TCP and 37777/TCP are relative to the RTSP streaming server, while 9090/TCP replies to ONVIF SOAP requests. The other ports (excluding 3232/TCP) result opened but the respective services do not reply to simple textual or HTTP request, many of them close the connection immediately.

“Speak, ‘friend’, and enter”

The most interesting port to obtain a direct access to a shell on the camera is 3232/TCP on which there is a Telnet daemon listening identified as “Busybox telnetd“. It replies with the prompt:

[root@wintermute ~]# telnet ipcamera.home 3232
Ambarella login:

The banner contains an interesting indication: a simple search on Google for “Ambarella” reveals that it is a Chinese IoT and camera producer.
I started to search online default passwords for Ambarella devices in order to find the right one for the root user. All the password found online do not correspond to the correct one and also a dictionary-based attack fails. I started to think to a brute-force attack with a charset composed only by lowers and digits limiting the maximum length to 8 chars. A simple calculation leads to ~2,8×10¹² tries but the speed of Hydra (and other bruteforce tools tested) is limited by the internal implementation of the telnet daemon, which slows down the speed to ~1200 tries/min per process. The time required to guess the root password (assuming the password is a maximum 8 chars string of only lower and digits) in an optimistic calculation (half of the cases with 64 parallel processes) is ~35 years… I tried unsuccessfully to look for some simple vulnerabilities of the web management interface but I had no success.
However I found an interesting, not normally shown, page which permits to configure settings for an RS232 interface. My camera does not have any VISIBLE external serial interface but probably (if the RS232 interface is not software emulated like SOL) it has some pins on the board not externally mapped. So I decided to obtain a physical access to the camera in order to find possible serial interface pins.

Here the internal structure of the camera (click on images to enlarge them).

“One UART to bring them all”

I suppose that, as the majority of modern embedded CPUs, the camera’s CPU (an Ambarella A1550 for which I’m not able to find any documentation online) has an UART interface on which I hope to find an open console, at least during the boot phase. Problem: the motherboard does not have labels around test and internal pins. Supposing the system uses the UART at least for sending logs during the boot phase I used a digital oscilloscope in order to identify the pattern of an UART communication on a “candidate” pin.
I focused my attention on two couples of pins: one couple is part of the soldered connector related to the Ethernet interface which reports “La” and “Lb” labels. The Ethernet cable in my camera is not wired to these pins so they are good candidates for a non-mapped RS232 interface. I tried to monitor both the pins during the boot process but none of them report activity.
The other couple of pins is located the border of the motherboard: when listening one of them during the boot process with the oscilloscope I received this signal waveform:

Signal waveform from one of the pins on the border of the motherboard.

Yes! It’s (most likely) an UART TX signal! There is an UART interface active on the motherboard but how can I identify the RX pin? In order to identify it I can send data through a candidate pin and check the effects on TX pin. I used my BusPirate3  which permits to send and receive data and analyze different digital protocols (as UART, SPI, I2C etc.). I connected it to a pin and opened a serial console on /dev/ttyACM0.
Problem: which are the connection parameters (speed in Baud, number of data bits per packet, parity, stop bit)? In theory I can deduce all of them directly from the captured waveform of the TX pin but I will try a common used set of them: 8 data bits, no parity, 1 bit of stop (8N1). The speed is easily calculable by the waveform: a 0 bit duration is ~10us so if we suppose 1 Baud = 1 symbol/sec = 1bit/sec (in case of digital communications) we obtain ~100000 Baud which is near to the standardized speed of 115200 Baud. Using this set of parameters the terminal shows, during the boot, the log of the kernel initialization.
Now that the connection parameters have been correctly identified I can try to identify the RX pin. I connected different RX candidate pins to the BusPirate and pushing ENTER continuously I waited for the logging console to display them on the screen. Finally I found the correct one. Now I have a bidirectional UART with the camera.

Amboot: the Ambarella bootloader

Here (a part of ) the log of the boot (XXXXXX replaces sensible information, the highlighted rows are important for the next analysis):

spinor flash ID is 0xXXXXXXXX
###gpio init###
flspinor addr = 0x00200000, size = 0x00DF0000
flspinor addr = 0x00050000, size = 0x001B0000
flspinor addr = 0x00040000, size = 0x00010000
flspinor addr = 0x00010000, size = 0x00030000
flspinor addr = 0x00000000, size = 0x00010000
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 3.10.73 (robot@dev-ubuntu-14) (gcc version 4.9.1 20140625 (prerelease) (crosstool-NG - Ambarella Linaro Multilib GCC [CortexA9 & ARMv6k] 2014.06) ) #12 PREEMPT Thu Mar 3 18:31:51 CST 2016
[    0.000000] CPU: ARMv7 Processor [414fc091] revision 1 (ARMv7), cr=10c53c7d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] Machine: Ambarella S2L (Flattened Device Tree), model: Ambarella S2LM Kiwi Board
[    0.000000] Memory policy: ECC disabled, Data cache writeback
[    0.000000] Ambarella:      AHB = 0xe0000000[0xe0000000],0x01000000 0
[    0.000000] Ambarella:      APB = 0xe8000000[0xe8000000],0x01000000 0
[    0.000000] Ambarella:      PPM = 0x00000000[0xdfe00000],0x00200000 9
[    0.000000] Ambarella:      AXI = 0xf0000000[0xf0000000],0x00030000 0
[    0.000000] Ambarella:    DRAMC = 0xdffe0000[0xef000000],0x00020000 0
[    0.000000] Kernel command line: console=ttyS0 root=/dev/mtdblock4 rw rootfstype=jffs2 init=/linuxrc
[    0.000000] Memory: 123484k/123484k available, 5540k reserved, 0K highmem
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
[    0.000000]     fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)
[    0.000000]     vmalloc : 0x88000000 - 0xff000000   (1904 MB)
[    0.000000]     lowmem  : 0x80000000 - 0x87e00000   ( 126 MB)
[    0.000000]     modules : 0x7f000000 - 0x80000000   (  16 MB)
[    0.000000]       .text : 0x80008000 - 0x803cb20c   (3853 kB)
[    0.000000]       .init : 0x803cc000 - 0x803ed94c   ( 135 kB)
[    0.000000]       .data : 0x803ee000 - 0x80419688   ( 174 kB)
[    0.000000]        .bss : 0x80419688 - 0x80439434   ( 128 kB)
[    0.000000] NR_IRQS:240
[    0.000000] sched_clock: 32 bits at 54MHz, resolution 18ns, wraps every 79536ms
[    0.000000] Console: colour dummy device 80x30
[    0.000000] console [ttyS0] enabled
[    0.355072] ambarella-pinctrl e8009000.pinctrl: Ambarella pinctrl driver registered
[    0.363147] ambarella-gpio gpio.0: Ambarella GPIO driver registered
[    0.373584] bio: create slab <bio-0> at 0
[    0.378703] ambarella-dma e0005000.dma: Ambarella DMA Engine
[    0.384888] ambarella-spi e0020000.spi: master is unqueued, this is deprecated
[    0.392992] ambarella-spi e0020000.spi: ambarella SPI Controller 0 created
[    0.401557] ambarella-i2c e8003000.i2c: Ambarella I2C adapter[0] probed!
[    1.899961] ambarella-i2c e8007000.i2c: No ACK from address 0xe8, 0:0!
[    1.906478] pca953x 2-0074: failed reading register
[    1.911366] pca953x: probe of 2-0074 failed with error -16
[    1.916844] ambarella-i2c e8007000.i2c: Ambarella I2C adapter[2] probed!
[    1.924257] Switching to clocksource ambarella-cs-timer
[    1.936633] ambarella-sd e0002000.sdmmc0: Slot0 use bounce buffer[0x87720000<->0x07920000]
[    1.944930] ambarella-sd e0002000.sdmmc0: Slot0 req_size=0x00020000, segs=32, seg_size=0x00020000
[    1.953798] ambarella-sd e0002000.sdmmc0: Slot0 use ADMA
[    2.029591] ambarella-sd e0002000.sdmmc0: 1 slots @ 50000000Hz
[    2.109563] ambarella-adc e801d000.adc: Ambarella ADC driver init
[    2.117275] jffs2: version 2.2. (NAND) 2001-2006 Red Hat, Inc.
[    2.154084] e8005000.uart: ttyS0 at MMIO 0xe8005000 (irq = 9) is a ambuart
[    2.162064] brd: module loaded
[    2.167871] loop: module loaded
[    2.171327] Ambarella read-only mtdblock
[    2.175458] 5 ofpart partitions found on MTD device amba_spinor
[    2.181435] Creating 5 MTD partitions on "amba_spinor":
[    2.186676] 0x000000000000-0x000000010000 : "bst"
[    2.192185] 0x000000010000-0x000000040000 : "bld"
[    2.197558] 0x000000040000-0x000000050000 : "ptb"
[    2.202998] 0x000000050000-0x000000200000 : "pri"
[    2.208365] 0x000000200000-0x000000ff0000 : "lnx"
[    2.213961] SPI NOR Controller probed
[    2.269504] libphy: Ambarella MII Bus: probed
[    2.273882] mdio_bus e000e000.etherne: /ahb@e000000 /ethernet@e000e000/phy@0 has invalid PHY address
[    2.283035] mdio_bus e000e000.etherne: scan phy phy at address 0
[    2.289659] mdio_bus e000e000.etherne: registered phy phy at address 0
[    2.296177] ambarella-eth e000e000.ethernet: Ethernet PHY[0]: 0x00221513!
[    2.303569] ambarella-eth e000e000.ethernet: MAC Address[XX:XX:XX:XX:XX:XX].
[    2.310843] ambarella-rtc e8015000.rtc: =====RTC ever lost power=====
[    2.339195] mmc0: error -84 whilst initialising SD card
[    2.389750] ambarella-rtc e8015000.rtc: rtc core: registered rtc ambarella as rtc0
[    2.517360] mmc0: error -84 whilst initialising SD card
[    2.707332] mmc0: error -84 whilst initialising SD card
[    2.897340] mmc0: error -84 whilst initialising SD card
[    4.302928] VFS: Mounted root (jffs2 filesystem) on device 31:4.
[    4.309119] devtmpfs: mounted
[    4.312341] Freeing unused kernel memory: 132K (803cc000 - 803ed000)
/etc/init.d/S12udev: /etc/ambarella.conf: line 1: export: not found

The CPU is an ARMv7 processor, the log shows the kernel initialization however the UART connection is only a console not a shell so I can’t access, for the moment, to the system. A very simple way to subvert this problem is to change the boot parameters. The cmdline used by the bootloader is

console=ttyS0 root=/dev/mtdblock4 rw rootfstype=jffs2 init=/linuxrc

so if I was able to change it to

console=ttyS0 root=/dev/mtdblock4 rw rootfstype=jffs2 init=/bin/sh

I would obtain a root access via console.
But I need to interrupt the boot sequence in order to access to bootloader and change the cmdline. I didn’t know what type of bootloader is present (arguably not UBoot) and I didn’t know how to stop the boot sequence.
I start to analyze the the bootlog in order to identify some useful information. The first lines of the boot log are:

spinor flash ID is 0xXXXXXXXX
###gpio init###
flspinor addr = 0x00200000, size = 0x00DF0000
flspinor addr = 0x00050000, size = 0x001B0000
flspinor addr = 0x00040000, size = 0x00010000
flspinor addr = 0x00010000, size = 0x00030000
flspinor addr = 0x00000000, size = 0x00010000
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Linux version 3.10.73 (robot@dev-ubuntu-14) (gcc version 4.9.1 20140625 (prerelease) (crosstool-NG - Ambarella Linaro Multilib GCC [CortexA9 &amp; ARMv6k] 2014.06)

so the lines before
[ 0.000000] Booting Linux on physical CPU 0x0
are generated by the bootloader. Searching them on Google I found out that they are generated by Amboot, the Ambarella boot loader. Furthermore it is possible to interrupt the boot sequence and enter into a boot menu simply pushing and holding ENTER while you power on the board. This is the Amboot menu:

spinor flash ID is 0xXXXXXXXX

             ___  ___  _________                _   
            / _ \ |  \/  || ___ \              | |  
           / /_\ \| .  . || |_/ /  ___    ___  | |_ 
           |  _  || |\/| || ___ \ / _ \  / _ \ | __|
           | | | || |  | || |_/ /| (_) || (_) || |_ 
           \_| |_/\_|  |_/\____/  \___/  \___/  \__|
Amboot(R) Ambarella(R) Copyright (C) 2004-2014
Boot From: SPI NOR 
SYS_CONFIG: 0x3000404B POC: 101
Cortex freq: 600000000
iDSP freq: 216000000
Dram freq: 528000000
Core freq: 216000000
AHB freq: 108000000
APB freq: 54000000
UART freq: 24000000
SD freq: 50000000
SDIO freq: 50000000
SDXC freq: 60000000

The Amboot menu permits to execute various task including loading an external image via TFTP server or via SD card. I used other two features: the possibility to load the system with a different command line and the “spinor” command (no, not this spinor4 but SPI NOR :D). Booting using the modified cmdline:
boot console=ttyS0 root=/dev/mtdblock4 rw rootfstype=jffs2 init=/bin/sh
I obtained

BusyBox v1.22.1 (2014-07-03 15:30:22 CST) multi-call binary
Enter 'help' for a list of built-in commands.
# id
uid=0(root) gid=0(root) groups=0(root)

Got it!

“That’s unsalted crypt(), baby. The unsalted crypt()! And there’s nothing you can do about it. Nothing!”

In order to crack the root password, I dumped /etc/shadow

# cat /etc/shadow

The hash of the root password has been generated using the DES-based schema which is not salted (it is possible to deduce it by observing the hash structure: it does not start by ‘$’ or ‘_’). Furthermore in the “traditional” DES-based crypt() hashes only the first 8 characters of the inserted password. I tried directly a brute-force attack using John with the same chars choice used with Hydra against the telnet daemon.

[root@wintermute ~]# john --incremental:LowerNum shadow
Loaded 1 password hash (descrypt, traditional crypt(3) [DES 128/128 SSE2-16])
Warning: MaxLen = 13 is too large for the current hash type, reduced to 8
Press 'q' or Ctrl-C to abort, almost any other key for status
nusiuh           (root)
Use the "--show" option to display all of the cracked passwords reliably
Session completed

It’s very easy to recover it from the hash, it took ~2 min on an Intel i5 with a default not optimized installation of John, the password is the camera brand name reversed :/ what wasted effort…


I rebooted the camera and entered again into the Amboot menu in order to use the command “spinor read START_ADDR STOP_ADDR” which shows the content of a section of the SPI NOR flash. I wrote a simple Python script able to send the spinor read command to the bootloader, capture the output and write it into a file and so I’m able reconstruct a 1:1 image of the entire flash.

A simple “binwalk” on the flash dump reveals its structure

202100        0x31574         CRC32 polynomial table, little endian
205337        0x32219         Copyright string: "Copyright (C) 2004-2014"
217398        0x35136         Unix path: /hs/sdr12/sdr25/sdr50/sdr104/ddr50
2097152       0x200000        JFFS2 filesystem, little endian
2752592       0x2A0050        Zlib compressed data, compressed
2754332       0x2A071C        Zlib compressed data, compressed
2755496       0x2A0BA8        Zlib compressed data, compressed
2758312       0x2A16A8        Zlib compressed data, compressed
2761820       0x2A245C        Zlib compressed data, compressed
2765576       0x2A3308        Zlib compressed data, compressed
2769104       0x2A40D0        Zlib compressed data, compressed
2770272       0x2A4560        Zlib compressed data, compressed
2773036       0x2A502C        Zlib compressed data, compressed

There are some JFFS2 filesystem headers (here it is shown only the first one) and a lot of “Zlib compressed data“, probably the native transparent compression method of the filesystem5. The dump contains all the partitions of the camera in a packed way, however I had a map of the partitions of the SPI NOR from the dmesg output, so I could “cut” the dump file using dd utility and reconstruct each partition.
As shown in the kernel log the camera has 5 partitions (their content is determined analysing the contained strings (or using binwalk), or in the case of bst and ptb the hex content):

0x000000-0x010000 : bst; Bootstrap partition (?)
0x010000-0x040000 : bld; Bootloader (Amboot)
0x040000-0x050000 : ptb; Partition table (?)
0x050000-0x200000 : pri ; Linux kernel ARM boot executable zImage (little-endian)
0x200000-0xff0000 : lnx; OS in a JFFS2 filesystem

In order to access to the root filesystem I needed to mount the lnx partition however the JFFS2 filesystem is a special type of filesystem, which does not permit to be mounted simply on a loop device. JFFS2 is specified designed to operate on MTD device: a device file, created by a module of the Linux kernel, able to directly interact with the underling flash memory device. It is possible to emulate a MTD device using the RAM with the mtdram module:

[root@wintermute ~]# modprobe mtdram
modprobe: FATAL: Module mtdram not found in directory /lib/modules/4.17.14-202.fc28.x86_64

DAMN! Fedora 28 does not have this module6 compiled for the default kernel release (07/04/2019, neither Fedora 30 has it…) and I was too lazy to recompile the kernel for only one module so I used an Ubuntu VM. On the Ubuntu VM:

root@ubuntu:~# modprobe mtdram total_size=32768 erase_size=64
root@ubuntu:~# modprobe mtdblock
root@ubuntu:~# dd if=lnx of=/dev/mtdblock0
root@ubuntu:~# mount -t jffs2 /dev/mtdblock0 /mnt

Here the erase_size block size is a fundamental parameter: if you use the wrong one you are not able to correctly mount the JFFS2 filesystem obtaining errors from the kernel module or inconsistent and missing data. In order to determine the correct erase_size you need the datasheet of the original SPI NOR flash (or obtain it from the camera /proc/mtd). I used the value extracted from the datasheet (the explanation of why I didn’t use the value from /proc/mtd will be cleared up). The datasheet of MX25L12835F7 SPI NOR flash contains (pages 53 and 54) two different values for the erase_size: 32K and 64K. I tried both, but only 64K returns the correct alignment. And finally:

root@ubuntu:~# ls /mnt
bin    dev  home  lib      mnt  proc  run   sys  usr
debug  etc  ipnc  linuxrc  opt  root  sbin  tmp  var
“That’s all folks!” (DAMN!)

Ok, after this (wasted) effort I have an image of the root filesystem and I can start to analyse (and emulate using qemu) all non standard binaries, libraries and kernel module contained in it and, maybe, find some vulnerability in them.
But… why I don’t read directly the erase_size from /proc/mtd or dump the partitions using dd if=/dev/mtdblockX of=/tmp/blkX directly on the powered-on camera? Because Amboot has a bug which corrupts the partitions if you interact with them and do not properly reboot the camera with the “reboot” command. So after the dump I have improperly rebooted the camera and at the moment it does not boot anymore (the kernel partition is corrupted and Amboot refuses to turn on Ethernet interface so no TFTP…) I will reflash the content of the partition using a SPI programmer as soon as the programming clip arrives.