Collateral Software
A blog by fschuindt

Table of Contents (All posts)

Encrypted FM radio: Using the Baofeng UV5R handheld transceiver to transmit and receive elliptic-curve encrypted messages with the AX.25 packet radio protocol over UHF (Modemless)

RX Cable

Introduction

This text showcases a PoC on transmitting Secp256k1 elliptic-cuve encrypted digital text messages over frequency-modulated voice radio within the UHF spectrum, only for educational purposes. For modern and/or practical applications on the same band, please check LoRa, FSK and ASK.

It also offers an alternative (modemless) approach to the one displayed at the Computerphile’s video Packet Radio (Post Apocalyptic Internet?).

Here are the steps:

  1. Generate two ECC key-pairs (Alice and Bob).
  2. Encrypt a digital text message from Alice to Bob.
  3. Convert the encrypted message into a .wav file.
  4. Transmit the .wav audio over FM UHF (Digital-to-Analog).
  5. Record the transmission at another station into a new .wav file (Analog-to-Digital).
  6. Convert the new recorded .wav back into the encrypted message string.
  7. Decrypt the message.

RX Station

RX Station

1. Generate two ECC key-pairs (Alice and Bob)

The following OpenSSL commands should create two key-pairs alongside their shared secrets.

# Generate private key files
openssl ecparam -name secp256k1 \
  -genkey -noout -out alice-private-key.pem

openssl ecparam -name secp256k1 \
  -genkey -noout -out bob-private-key.pem

# Generate public key files
openssl ec -in alice-private-key.pem \
  -pubout -out alice-public-key.pem

openssl ec -in bob-private-key.pem \
  -pubout -out bob-public-key.pem

# Generate shared secret files
openssl pkeyutl -derive \
  -inkey alice-private-key.pem \
  -peerkey bob-public-key.pem \
  -out alice-shared-secret.bin

openssl pkeyutl -derive \
  -inkey bob-private-key.pem \
  -peerkey alice-public-key.pem \
  -out bob-shared-secret.bin

It should create the following files:

alice-private-key.pem
alice-public-key.pem
alice-shared-secret.bin

bob-private-key.pem
bob-public-key.pem
bob-shared-secret.bin

2. Encrypt a digital text message from Alice to Bob

Now, on the same directory, let’s write the message we want to encrypt. I’m going to set it as secret-message.txt.

secret-message.txt

New car, caviar, four star, daydream.
Think I'll buy me a football team.

And to encrypt it:

openssl enc -aes256 -base64 -k \
  $(base64 -w 0 alice-shared-secret.bin) \
  -pbkdf2 -e -in secret-message.txt \
  -out secret-message-cipher.txt

It will result in a secret-message-cipher.txt file, with the encrypted message.

Mine looks like so:

U2FsdGVkX1+mhD4zEq13II1kO7OBmPw7UuPWnedqSSC+9jJ/aXCFm6kF8bEtyBa0
4WUOk0Bawux+jKvQmSRRKvWNaRwdlTaWsZJyx9lWEqbubVY3FFWDCJ6L5A3K3kg6

3. Convert the encrypted message into a .wav file

For that, we’re going to use the AX.25 protocol (See also ax25.net).

In the past, a physical device (the modem) was needed in order to convert the digital signal into analog AX.25 packets (and vice versa). However, today we can find software-based modem implementations like Dire Wolf. Those allow us to perform RF AX.25 communications without any physical devices other than the radio units and their respective cables.

To send messages, Dire Wolf uses the TAPR TNC2’s monitoring message format (See also 1. This comment 2. The KISS protocol). Simplifying for our needs, we can use the following format adaptation:

SOURCE>DESTINATION:payload

While SOURCE and DESTINATION are reserved for stations call signs, for the purposes of this PoC we’re going to use ALICE and BOB. The payload should be, in this case, the content of the secret-message-cipher.txt file.

Example:

ALICE>BOB:U2FsdGVkX1+mhD4zEq13II1kO7OBmPw [...]

The following command should instruct Dire Wolf’s gen_packets program to create a message-to-broadcast.wav file with the encrypted message modulated as AX.25 packets:

echo -n 'ALICE>BOB:'$(cat secret-message-cipher.txt) \
  | gen_packets -o message-to-broadcast.wav -

That file should now be transmitted over the radio. Fun fact: If you listen to it, it sounds just like a Dial-Up connection.

message-to-broadcast.wav file:

4. Transmit the .wav audio over FM UHF (Digital-to-Analog)

I’m going to use 1W TX power at the 462.587MHz (UHF) frequency, which is FRS’ Channel 2. And of course, the Baofeng uses narrow FM.

AFAIK, regulations-wise, AX.25 packets can not be sent within the FRS bands. However, I’m going to do it just for the sake of this demonstration. Moreover, here in my region all these channels are being used by all kinds of pirates, including digital. In some sense, I think this is good, as it maintains the hobby alive. And we all expect a lot of noise within UHF anyway. But here goes a formal disclaimer.

4.1. Warning

Before going any further, I must warn you that doing so might be illegal on your country. Please consult local regulations. Be sure to use the correct part of the spectrum and to use the correct power settings. Ideally, you should have a local license for transmitting.

With that, we continue.

4.2. Cable modification

Every Baofeng UV-5R comes with a PTT, microphone and speaker cable, just like this one:

I modified two of these cables, one to act as the TX cable and the other to act as the RX cable. However, if you are interested into full-duplex capabilities, you should modify your cables using the TX schematics (follows), which will support both TX and DX.

4.2.1 TX Cable

TX Cable

Schematics (Open Source project): Schematics

And I replaced both the microphone and the speaker with a p2-stereo male cable each. Even though the signal is mono, those were the pieces I had lying around, as most of this was made with recycled parts.

I also replaced the PTT button for a slightly bigger one.

4.2.2 RX Cable

RX Cable

Schematics: RX Cable Drawing

4.4 Transmit

Here follows a simple diagram of what’s happening. (Open here for rendering it)

The TX cable is connecting one UV-5R’s microphone to ALICE’s computer p2 audio output. The RX cable connects the other UV-5R’s speaker to BOB’s computer p2 line input. With the two radios turned on and tuned to the same frequency, BOB’s computer starts to record the audio signal from the line input. Then on ALICE, the PTT is pressed, and the message-to-broadcast.wav is played, which will be transmitted by the radio, then recorded by BOB’s computer.

5. Record the transmission at another station into a new .wav file (Analog-to-Digital)

On BOB’s computer, the recording results in a recorded-broadcast.wav file:

6. Convert the new recorded .wav back into the encrypted message string

Now we can use Dire Wolf’s atest program to demodulate it back into text:

$ atest recorded-broadcast.wav

44100 samples per second.  16 bits per sample.  2 audio channels.
1314816 audio bytes in file.  Duration = 7.5 seconds.
Fix Bits level = 0
Channel 0: 1200 baud, AFSK 1200 & 2200 Hz, E, 44100 sample rate.
Channel 1: 1200 baud, AFSK 1200 & 2200 Hz, E, 44100 sample rate.

DECODED[1] 0:03.587 ALICE audio level = 0(0/0)
[0] ALICE>BOB:U2FsdGVkX1+mhD4zEq13II1kO7OBmPw7UuPWnedqSSC+9jJ/aXCFm6kF8bEtyBa0 4WUOk0Bawux+jKvQmSRRKvWNaRwdlTaWsZJyx9lWEqbubVY3FFWDCJ6L5A3K3kg6


1 from v1.wav
1 packets decoded in 0.093 seconds.  80.6 x realtime

And sure enough, the text is there:

[0] ALICE>BOB:U2FsdGVkX1+mhD4zEq13II1kO7OBmPw7UuPWnedqSSC+9jJ/aXCFm6kF8bEtyBa0 4WUOk0Bawux+jKvQmSRRKvWNaRwdlTaWsZJyx9lWEqbubVY3FFWDCJ6L5A3K3kg6

7. Decrypt the message.

atest added a white space instead of a \n, but this is no problem:

U2FsdGVkX1+mhD4zEq13II1kO7OBmPw7UuPWnedqSSC+9jJ/aXCFm6kF8bEtyBa0
4WUOk0Bawux+jKvQmSRRKvWNaRwdlTaWsZJyx9lWEqbubVY3FFWDCJ6L5A3K3kg6

That’s the exact same text. Now, just decrypt it:

openssl enc -aes256 -base64 \
  -k $(base64 -w 0 bob-shared-secret.bin) \
  -pbkdf2 -d -in extracted-message.txt \
  -out result.txt \
  && cat result.txt

Which outputs:

New car, caviar, four star, daydream.
Think I'll buy me a football team.

Great!

  • Cover picture: Midjourney prompt: illuminated manuscript, medieval art, pre-renaissance, a radio operator using a computer –v 5 –aspect 7:4 –seed 722
  • Thanks to the YouTube user “Karst & Coffee” for mentioning about software modems.

Mineral Moon with the Askar FRA400 and a Canon T3i

I took this picture in early November 2022, but only now I’m posting it. It’s the best Moon picture I took so far, the Askar FRA400 really speaks for itself.

Author: Fernando Schuindt
License: CC BY 4.0
Camera: BCF-1 modified Canon T3i
Lens/Telescope: Askar FRA400 400mm f/5.6 quintuplet apochromat
Composition: 98x Light Frames, ISO 100, 1/1000”
Other Gear: Meade LX85 GEM unguided (Two-star basic alignment)
Processing: Capturing with APT, pre-processing with PIPP, stacking with AutoStakkert, stretching with RegiStax 6 and editing with Photoshop
Location Name: Aracaju - Sergipe, Brazil
Location Aprox. Coordinates: 10°57’30.0”S 37°02’30.0”W
Timestamp (Local Time): 10-11-2022 from 23:30:51h to 23:31:08h
Timezone: UTC-3 (No daylight saving time)
Theme: Moon
Outdoor temp.: 27° C
Outdoor humidity: 77%
Backup Images (On Google Photos): https://photos.app.goo.gl/u8faTtVy5mKNXz2H7

(Click to zoom)

Picture

48x 12s ISO 1600 on M42

Finally, a brief moment of clear night sky. I’m hoping things will start to get better up to the summer. But anyway, this is my first stacked picture using the FRA400 telescope. No guiding, really crude alignment and super short exposures. But even so, I’m happy with the result. Looking forward to improving data gathering and processing techniques.

Author: Fernando Schuindt
License: CC BY 4.0
Camera: BCF-1 modified Canon T3i
Lens/Telescope: Askar FRA400 400mm f/5.6 quintuplet apochromat
Composition: 48x Light Frames 12” (9.6’ total) ISO 1600, 25x dark, 25x flat, 25x bias
Processing: Stacking with DeepSky Stacker, Photoshop Levels adjustments and Camera Raw Filter
Location Name: Aracaju - Sergipe, Brazil
Bortle Scale: 7
Location Aprox. Coordinates: 10°57’30.0”S 37°02’30.0”W
Timestamp (Local Time): 10-23-2022 from 00:54h to 01:16h
Timezone: UTC-3 (No daylight saving time)
Outdoor temp.: 28° C
Outdoor humidity: 73%
Theme: M42
Center (RA, hms): 05h 35m 47.375s
Center (Dec, dms): -05° 44’ 44.402”
Legacy Surveys sky browser: Click here
Other Gear: Meade LX85 GEM unguided (Two-star basic alignment)
Astrometry: https://nova.astrometry.net/user_images/6636562#annotated
Backup Images (On Google Photos): https://photos.app.goo.gl/YWpmThGFRFDU1hBt9

(Click to zoom)

Picture

Annotated: Picture with notes

Jupiter and the Galilean moons with the Kugelblitz 210mm f/6.62 handmade telescope, a ZWO ASI462MC, a Meade LX85 GEM and an Optolong UV/IR-cut filter.

For long, I was willing to try to use the big 1390mm focal length ATM scope into my LX85 mount. I knew it was sketchy, but it being well under the maximum mount payload, I decided to give it a try. Also, these days with Jupiter so close I thought it would be a good moment. So I called my buddy Leandro, and we began to set up everything. Many thanks to him.

It is fun to see that huge scope with that GEM. The wind shook the hell out of it, but it was a nice experience.

Author: Fernando Schuindt
License: CC BY 4.0
Lens/Telescope: Kugelblitz handmade telescope (210mm aperture, 1390mm focal length, f/6.62, Newtonian)
Camera: ZWO ASI462MC (Color)
Mount: Meade LX85 GEM
Other Gear: Optolong UV/IR-cut filter, Redlab x86 homemade weather-resistant computer
Distance: 3.956 AU
Magnitude: -2.93
Location Name: Aracaju - Sergipe, Brazil
Location Aprox. Coordinates: 10°57’30.0”S 37°02’30.0”W
Processing: SharpCap, PIPP, AutoStakkert, RegiStax 6, Photoshop
Full Resolution Images: https://photos.app.goo.gl/zMKvb51DQPHMqVAS7

Theme: Jupiter and the Galilean moons

Jupiter with notes (Click to open full resolution)

Theme: Jupiter and the Galilean moons
Composition: 2000 frames captured (11.8GB .avi file), 600 frames stacked
Timestamp (UTC-3): 30-09-2022 23:21:27

Jupiter with notes (Click to open full resolution)

Jupiter with notes (Click to open full resolution)

Theme: Jupiter and the Galilean moons (with Celestron 2x barlow lens)
Composition: 1262 frames captured (7.8GB .avi file), 450 frames stacked
Timestamp (UTC-3): 30-09-2022 23:29:55

Jupiter with notes (Click to open full resolution)

Jupiter with notes (Click to open full resolution)

Gear & Bonus

Jupiter with notes (Click to open full resolution)

Camera Settings

Without 2x barlow

[ZWO ASI462MC]
FrameType=Light
Pan=0
Tilt=0
Output Format=AVI files (*.avi)
Binning=1
Capture Area=1936x1096
Colour Space=RGB24
Temperature=36
High Speed Mode=On
Turbo USB=100(Auto)
Flip=None
Frame Rate Limit=Maximum
Gain=179
Exposure=3.2030ms
Timestamp Frames=Off
White Bal (B)=58
White Bal (R)=66
Brightness=95
Auto Exp Max Gain=300
Auto Exp Max Exp M S=30000
Auto Exp Target Brightness=110
Mono Bin=Off
Background Subtraction=Off
Planet/Disk Stabilization=Off
Banding Threshold=35
Banding Suppression=0
Apply Flat=None
Subtract Dark=None
Display Black Point=0
Display MidTone Point=0.521367521367521
Display White Point=1
Notes=
TimeStamp=2022-10-01T02:21:27.7696808Z
SharpCapVersion=4.0.8395.0
StartCapture=2022-10-01T02:21:27.7667114Z
MidCapture=2022-10-01T02:21:41.4127114Z
EndCapture=2022-10-01T02:21:55.0595507Z
Duration=27.293s
FrameCount=2000
TimeZone=-3.00

With 2x barlow

[ZWO ASI462MC]
FrameType=Light
Pan=0
Tilt=0
Output Format=AVI files (*.avi)
Binning=1
Capture Area=1936x1096
Colour Space=RGB24
Temperature=35.3
High Speed Mode=On
Turbo USB=100(Auto)
Flip=None
Frame Rate Limit=Maximum
Gain=223
Exposure=9.0760ms
Timestamp Frames=Off
White Bal (B)=58
White Bal (R)=69
Brightness=80
Auto Exp Max Gain=300
Auto Exp Max Exp M S=30000
Auto Exp Target Brightness=110
Mono Bin=Off
Background Subtraction=Off
Planet/Disk Stabilization=Off
Banding Threshold=35
Banding Suppression=0
Apply Flat=None
Subtract Dark=None
Display Black Point=0
Display MidTone Point=0.521367521367521
Display White Point=1
Notes=
TimeStamp=2022-10-01T02:29:55.1520307Z
SharpCapVersion=4.0.8395.0
StartCapture=2022-10-01T02:29:55.1482424Z
MidCapture=2022-10-01T02:30:05.2302424Z
EndCapture=2022-10-01T02:30:15.3125747Z
Duration=20.164s
FrameCount=1262
TimeZone=-3.00

Total lunar eclipse with an ATM Dobsonian and a modified Canon T3i

During the night of May 15th 2022, I went to the SEASE event for the total lunar eclipse at the CCTECA Planetarium Galileu Galilei, in Aracaju - SE, Brazil. I brought with me my homemade 210mm f/6.62 Newtonian Dobsonian telescope and some equipments. The Moon’s FOV on the Canon T3i with that telescope is almost the entire frame, perfect for full-disk single frame photography. That was my first time photographing a lunar eclipse, so I wanted to keep things simple.

The event was really nice, a lot of people went to watch the Moon during its transformations for the night, and I spent a lot of time handling the telescope to the public. People of all ages saw the Moon through my telescope, and I was pleased to make some friends on the spot.

Many thanks to my friend Leandro who was with me all along and helped me during all the tasks, he was super. Also thanks to the SEASE for the space and reception.

Fun fact is that a few days after my picture appeared at the local news along with some others, you can watch it here: https://www.youtube.com/watch?v=CZ834A3z2Bc

Thanks to Augusto from SEASE for sending the pictures to them.

Author: Fernando Schuindt
License: CC BY 4.0
Camera: Modified Canon T3i (Baader BCF, by Jordan Patrick)
Camera Settings: ISO 1600~3200, 1/5~1/2”, f/6.62*
Lens/Telescope: Kugelblitz handmade telescope (210mm aperture, 1390mm focal length, f/6.62, Newtonian)
Processing: Capturing without software, Photoshop camera RAW filter, crop, resize and arrangement
Location Name: Aracaju - Sergipe, Brazil
Location Aprox. Coordinates: 10°56’39.0”S 37°03’21.0”W
Theme: Total lunar eclipse
Composition: 4 single frames, one for each moment of the eclipse
Timestamp (Local Time, UTC-3): 15-05-2022 00:29:18 ~ 16-05-2022 01:16:44
Full Resolution Images: https://photos.app.goo.gl/5dNkmo72mZEEFyir7

Result (Click to open full resolution)

And here goes a bonus picture I took from the event:

Picture of the event (Click to open full resolution)

My telescope is the one being observed at the middle.

My first Moon mosaic

Experimenting with mosaics I think that the IR-pass turned out to be a better approach. Sadly I have missed a bit from the Moon during the IR-pass capture.

Author: Fernando Schuindt
License: CC BY 4.0
Camera: ZWO ASI462MC (Color)
Lens/Telescope: Meade ETX90 (OTA-only) (90mm aperture, 1250mm focal length, f/13.88, Maksutov-Cassegrain)
GEM: Meade LX85
Other Gear: Redlab x86 homemade weather-resistant astronomy computer
Processing: Capturing with SharpCap, pre-processing with PIPP, stacking with AutoStakkert, stretching with RegiStax 6, editing with Photoshop and mosaic creation with Microsoft Image Composite Editor
Location Name: Aracaju - Sergipe, Brazil
Location Aprox. Coordinates: 10°58’31.0”S 37°04’26.0”W
Full Resolution Images: https://photos.app.goo.gl/6xuYX1aYLu6xFsim6

Theme: IR-pass waxing crescent Moon mosaic
Filters: ZWO IR 850nm pass filter
Composition: 8 video files (sections) for a total of 15.9GB
Timestamp (Local Time): 10-10-2021 19:52:26
Timestamp (UTC): 10-10-2021 22:52:26

IR (Click to open full resolution)

Theme: Waxing crescent Moon mosaic
Composition: 9 video files (sections) for a total of 53.6GB
Timestamp (Local Time): 10-10-2021 19:41:06
Timestamp (UTC): 10-10-2021 22:41:06

Light (Click to open full resolution)

Jupiter and the Galilean moons with the Kugelblitz handmade telescope and Saturn on a Meade ETX90

The wet season is still bringing lots of rain to my region, but once in a while there’s a good night sky and good opportunity to perform observations and also to take some pictures. As both Jupiter and Saturn are close to the zenith around midnight on a moonless night I thought I’d go for them.

I could say this is my first real planetary image using the ZWO ASI462MC color camera. I’ve performed the following experiments:

  • Jupiter, no barlow, ETX90
  • Jupiter, no barlow, Kugelblitz
  • Jupiter, 2x barlow, ETX90
  • Jupiter, 2x barlow, Kugelblitz
  • Saturn, no barlow, ETX90

I was focusing my attention over Jupiter, so the Saturn picture is really a bonus. The next time I’ll try Saturn, and I’ll also experiment using the ZWO 850nm IR-pass filter.

Even so the ETX90 was taking advantage of a LX85 equatorial mount against a Dobsonian mount of the Kugelblitz, the Kugelblitz images turned a lot more clear. I think that was mainly due to the f/6.62 of the Kugelblitz against the f/13.88 of the ETX90.

I’m also planning to build a Bahtinov mask to aid in focusing, as I was using manual/visual focusing, which makes it pretty hard to achieve good focus.

Author: Fernando Schuindt
License: CC BY 4.0
Camera: ZWO ASI462MC (Color)
Other Gear: Redlab x86 homemade weather-resistant astronomy computer
Location Name: Aracaju - Sergipe, Brazil
Location Aprox. Coordinates: 10°58’31.0”S 37°04’26.0”W
Full Resolution Images: https://photos.app.goo.gl/ZzZtCWdpbauMmmZz9

Theme: Jupiter and the Galilean moons with notes
Distance: 4.039 AU
Magnitude: -2.85
Lens/Telescope: Kugelblitz handmade telescope (210mm aperture, 1390mm focal length, f/6.62, Newtonian)
Composition: 2698 frames captured (16.7GB .avi file), 700 frames stacked
Processing: AutoStakkert for stacking, RegiStax 6 for stretching and Photoshop for processing
Timestamp (Local Time): 06-08-2021 22:19:21
Timestamp (UTC): 07-08-2021 01:19:21

Jupiter with notes (Click to open full resolution)

Theme: Jupiter and the Galilean moons without notes
Distance: 4.039 AU
Magnitude: -2.85
Lens/Telescope: Kugelblitz handmade telescope (210mm aperture, 1390mm focal length, f/6.62, Newtonian)
Composition: 2698 frames captured (16.7GB .avi file), 700 frames stacked
Processing: AutoStakkert for stacking, RegiStax 6 for stretching and Photoshop for processing
Timestamp (Local Time): 06-08-2021 22:19:21
Timestamp (UTC): 07-08-2021 01:19:21

Jupiter without notes (Click to open full resolution)

Theme: Saturn
Distance: 8.938 AU
Magnitude: 0.19
Lens/Telescope: Meade ETX90 (OTA-only) (90mm aperture, 1250mm focal length, f/13.88, Maksutov-Cassegrain)
Other Gear: Meade LX85 GEM (unguided, two-star alignment)
Composition: 1002 frames captured (5.94GB .avi file), 500 frames stacked
Processing: AutoStakkert for stacking, RegiStax 6 for stretching and Photoshop for processing
Timestamp (Local Time): 06-08-2021 23:06:00
Timestamp (UTC): 07-08-2021 02:06:00

Saturn (Click to open full resolution)

Gear

ETX90, LX85, 462MC and the redlab. (And Zig on the background) ETX90, LX85, 462MC and the redlab

Kugelblitz, 462MC and the redlab. Kugelblitz, 462MC and the redlab.

Inside the redlab. Inside the redlab

Setup pictures. Setup picture 1

Setup picture 2

Zig is always around s2. Setup picture 3

Camera Settings

Jupiter:

[ZWO ASI462MC]
Pan=0
Tilt=0
Output Format=AVI files (*.avi)
Binning=1
Capture Area=1936x1096
Colour Space=RGB24
Temperature=36
High Speed Mode=On
Turbo USB=100(Auto)
Flip=None
Frame Rate Limit=Maximum
Gain=114
Exposure=0.003776
Timestamp Frames=Off
White Bal (B)=63(Auto)
White Bal (R)=53
Brightness=102
Auto Exp Max Gain=300
Auto Exp Max Exp M S=30000
Auto Exp Target Brightness=130
Mono Bin=Off
Banding Threshold=35
Banding Suppression=0
Apply Flat=None
Subtract Dark=None
#Black Point
Display Black Point=0
#MidTone Point
Display MidTone Point=0.521367521367521
#White Point
Display White Point=1
Notes=
TimeStamp=2021-08-06T01:19:21.7457571Z
SharpCapVersion=3.2.6482.0

Saturn:

[ZWO ASI462MC]
Pan=0
Tilt=0
Output Format=AVI files (*.avi)
Binning=1
Capture Area=1936x1096
Colour Space=RGB24
Temperature=31.1
High Speed Mode=On
Turbo USB=100(Auto)
Flip=None
Frame Rate Limit=Maximum
Gain=264
Exposure=0.015104
Timestamp Frames=Off
White Bal (B)=57(Auto)
White Bal (R)=58
Brightness=152
Auto Exp Max Gain=300
Auto Exp Max Exp M S=30000
Auto Exp Target Brightness=130
Mono Bin=Off
Banding Threshold=35
Banding Suppression=0
Apply Flat=None
Subtract Dark=None
#Black Point
Display Black Point=0
#MidTone Point
Display MidTone Point=0.521367521367521
#White Point
Display White Point=1
Notes=
TimeStamp=2021-08-06T02:06:00.7942288Z
SharpCapVersion=3.2.6482.0

Waxing crescent Moon (26.47%) with the ZWO ASI462MC camera and a Mak90 telescope

Close up on the Moon using the ASI426MC camera with a 1250mm focal distance / 90mm aperture (f/13.88) telescope on top of a Meade LX85 computerized german equatorial mount.

Author: Fernando Schuindt
License: CC BY 4.0
Camera: ZWO ASI462MC
Lens/Telescope: Meade ETX 90 (OTA-only) (90mm aperture, 1250mm focal distance, f/13.88, Maksutov-Cassegrain)
Mount: Meade LX85 GEM (Two-star alignment).
Software: SharpCap 3.2, PIPP, AutoStakkert, RegiStax 6, Adobe Photoshop CC 2015.
Location Name: Aracaju - Sergipe, Brazil
Bortle Scale: 7
Location Aprox. Coordinates: 10°58’31.0”S 37°04’26.0”W
Timezone: UTC-3 (No daylight saving time)
Outdoor temp.: 26° C
Outdoor humidity: 98%
Backup Images (On Google Photos): https://photos.app.goo.gl/C6m7Ebpt6Qj83VSCA

(Click to zoom)

Picture 1

Composition: 1000 frames captured, stacked 63.
Timestamp (Local Time): 06-16-2021 19:38:22
Theme: 26.47% Moon

CameraSettings.txt:

[ZWO ASI462MC]
Pan=0
Tilt=0
Output Format=AVI files (*.avi)
Binning=1
Capture Area=1936x1096
Colour Space=RGB24
Temperature=35.3
High Speed Mode=On
Turbo USB=100(Auto)
Flip=None
Frame Rate Limit=120 fps
Gain=64
Exposure=0.032
Timestamp Frames=Off
White Bal (B)=65(Auto)
White Bal (R)=53
Brightness=78
Auto Exp Max Gain=300
Auto Exp Max Exp M S=30000
Auto Exp Target Brightness=130
Mono Bin=Off
Banding Threshold=35
Banding Suppression=0
Apply Flat=None
Subtract Dark=None
#Black Point
Display Black Point=0
#MidTone Point
Display MidTone Point=0.521367521367521
#White Point
Display White Point=1
Notes=
TimeStamp=2021-06-16T22:38:22.0496841Z
SharpCapVersion=3.2.6482.0

Picture 2

Composition: 1000 frames captured, stacked 70.
Timestamp (Local Time): 06-16-2021 20:09:05
Theme: 26.47% Moon
Other Gear: Celestron 2x barlow lens Multi-Coated, Optolong UV-IR Cut 1.25” filter.

CameraSettings.txt:

[ZWO ASI462MC]
Pan=0
Tilt=0
Output Format=AVI files (*.avi)
Binning=1
Capture Area=1936x1096
Colour Space=RGB24
Temperature=34.6
High Speed Mode=On
Turbo USB=100(Auto)
Flip=None
Frame Rate Limit=Maximum
Gain=364
Exposure=0.016
Timestamp Frames=Off
White Bal (B)=76(Auto)
White Bal (R)=53
Brightness=102
Auto Exp Max Gain=300
Auto Exp Max Exp M S=30000
Auto Exp Target Brightness=130
Mono Bin=Off
Banding Threshold=35
Banding Suppression=0
Apply Flat=None
Subtract Dark=None
#Black Point
Display Black Point=0
#MidTone Point
Display MidTone Point=0.521367521367521
#White Point
Display White Point=1
Notes=
TimeStamp=2021-06-16T23:09:05.1548647Z
SharpCapVersion=3.2.6482.0

Streaming video and audio of an USB webcam to multiple users of a website with SSL, basic authentication and in-video timestamps (FFmpeg, RTMP, NGINX, HLS and MPEG-Dash)

GIF

(Please don’t mind the mess)

Having an USB webcam connected to a Raspberry Pi 4, we’re going to use FFmpeg to securely (SSL and secret-key authentication) stream its video and audio (with CCTV-style timestamps) to a containerized NGINX server over RTMP/RTMPS protocol, and then use this same server to broadcast the stream to multiple users using both Apple HLS and MPEG-Dash. Users will be able to watch the stream on the browser, Android, iOS, Linux, Windows and MacOS.

Webcam

Architecture layout

This is useful if you’re willing to have any video source (file or live feed) streaming on the internet or network, maybe also host the stream on your own website (embedded as HTML) to display public events/places, all-sky-cameras, CCTV, science experiments, etc. Or anything that won’t match with some of the major streaming platform policies. Or something that’s designed to broadcast 24/7. The cases are numerous. Whatever the use-case be, I have applied here simple but efficient methods to limit who can stream, who can watch and also to end-to-end encrypt everything. In this article I’m going to demonstrate how I configured it all using common “of-the-shelf” tools, and I hope this helps you.

But before we begin, note that using this configuration I’m experiencing about 50 seconds of delay between what’s recorded and what’s played with all devices communicating over a 54mpbs Wifi network. But the biggest bottleneck I assume it’s NGINX processing the stream.

Preparing the NGINX Server

The most important part of this setup is the NGINX server. This server is going to receive the media stream from the Raspberry Pi, decrypt it, authenticate and serve it as both HLS and Dash. However NGINX won’t support RTMP media stream by default, we’ll have to compile it with the RTMP module. This process is straight-forward but tedious, so I’ve built a Docker image to do that for us. In case you want to do it manually for some reason you can follow the commands of the Dockerfile:

For you to set this container up, first clone the Docker image repository:

$ git clone https://github.com/fschuindt/nginx_rtmp_hls_dash.git

Then cd into it:

cd nginx_rtmp_hls_dash

Now you will need Docker and Docker Compose installed on your machine. Installing them is pretty simple, so take a minute to do so if you don’t have them already.

Build the image:

$ docker-compose build broadcaster

And spawn up the NGINX server:

$ docker-compose up broadcaster

The server is now ready to accept streams and to start broadcasting them.

This will leave NGINX running attached to the 4080, 4443, 4080 and 4936 ports on the Docker host machine. Check the repository documentation for more information on those ports. Or better yet, take a look at the nginx.conf file:

You can change this file as you see fit, just be sure to check the “Warnings” section of the repository documentation.

As you can see on the nginx.conf, we have a rudimentar secret-key based authentication. It’s enough for the purpose of this article and for the most simple usage of media streaming. Mind though that it only makes sense to use this authentication over SSL/TLS encryption, because the keys will be sent as plain text otherwise.

The configuration also describes a SSL and a plain-text endpoint for each connection. So you can choose between encrypted and unencrypted on all steps.

Sending webcam audio and video to the NGINX server

Now that the server is up and running, we need to SSH into the Raspberry Pi (or any computer with an USB webcam). In my case the Pi has a generic USB webcam connected to it. Our goal is to find this webcam device and to use FFmpeg to start a RTPM and RTPMS audio/video stream to the NGINX server over the network. Be sure to have FFmpeg installed on your Pi/computer, this should be simple to do.

Also mind that here I’m going to be describing steps for a Linux computer. Although FFmpeg is multi-platform, the way the video data input is harvested may change depending on the OS. I’m going to use Video4Linux. On the repository README.md you will find a more simple example for streaming .mp4 files.

The first step is to find the webcam device:

$ v4l2-ctl --list-devices

bcm2835-codec-decode (platform:bcm2835-codec):
	/dev/video10
	/dev/video11
	/dev/video12
	/dev/media0

bcm2835-isp (platform:bcm2835-isp):
	/dev/video13
	/dev/video14
	/dev/video15
	/dev/video16
	/dev/media1

GENERAL WEBCAM: GENERAL WEBCAM (usb-0000:01:00.0-1.3):
	/dev/video0
	/dev/video1
	/dev/media2

This output tells me that /dev/video0 is the webcam.

This webcam have a built-in microphone and we also want to stream the audio, so we need to find its audio interface:

$ arecord -L

null
    Discard all samples (playback) or generate zero samples (capture)
samplerate
    Rate Converter Plugin Using Samplerate Library
speexrate
    Rate Converter Plugin Using Speex Resampler
jack
    JACK Audio Connection Kit
oss
    Open Sound System
pulse
    PulseAudio Sound Server
speex
    Plugin using Speex DSP (resample, agc, denoise, echo, dereverb)
upmix
    Plugin for channel upmix (4,6,8)
vdownmix
    Plugin for channel downmix (stereo) with a simple spacialization
default
    Default ALSA Output (currently PulseAudio Sound Server)
usbstream:CARD=Headphones
    bcm2835 Headphones
    USB Stream Output
sysdefault:CARD=WEBCAM
    GENERAL WEBCAM, USB Audio
    Default Audio Device
front:CARD=WEBCAM,DEV=0
    GENERAL WEBCAM, USB Audio
    Front output / input
usbstream:CARD=WEBCAM
    GENERAL WEBCAM
    USB Stream Output

It’s the one set as the default: sysdefault:CARD=WEBCAM.

We also need to discover the supported resolutions for the webcam:

$ ffmpeg  -hide_banner -f video4linux2 -list_formats all -i /dev/video0

[video4linux2,v4l2 @ 0xaaaac7504320] Compressed:       mjpeg :          Motion-JPEG : 1920x1080 1440x1080 1280x720 800x600 800x480 720x480 640x480 640x360 480x270 320x240 176x144
[video4linux2,v4l2 @ 0xaaaac7504320] Raw       :     yuyv422 :           YUYV 4:2:2 : 1280x720 800x600 800x480 720x480 640x480 640x360 480x270 320x240 176x144

I think 1280x720 is a good choice for this experiment.

Now a quick side note: I decided to play with webcam streaming to help me in a weather project for astronomy data acquisition, so for me a CCTV-style timestamp and the ability to write text on the video was essential. Luckily FFmpeg has it all covered (check the drawtext option on the next command).

As my laptop’s network IP (running the NGINX) is 192.168.1.192, this is how I composed my RTMPS webcam streaming with in-video timestamps, from the Raspberry Pi:

ffmpeg \
    -f video4linux2 -framerate 25 -video_size 1280x720 -i /dev/video0 \
    -f alsa -ac 2 -i sysdefault:CARD=WEBCAM \
    -c:v libx264 -b:v 1600k -preset ultrafast \
    -x264opts keyint=50 -g 25 -pix_fmt yuv420p \
    -c:a aac -b:a 128k \
    -vf "drawtext=fontfile=/usr/share/fonts/dejavu/DejaVuSans-Bold.ttf: \
text='CLOUD DETECTION CAMERA 01 UTC-3 %{localtime\:%Y-%m-%dT%T}': [email protected]: fontsize=16: x=10: y=10: box=1: boxcolor=black: boxborderw=6" \
    -f flv "rtmps:192.168.1.192:4936/live/cam1?streamkey=5f3e32f3bad0"

Note the rtmps:192.168.1.192:4936/live/cam1?streamkey=5f3e32f3bad0 address, sending the RTMPS to the NGINX server.

Be sure to read the repository README.md to learn more about the address format and the stream-key.

After running it, the RPi4 is already streaming the webcam audio and video to the NGINX server.

Watching it: HLS or MPEG-Dash?

Many articles go deep into the difference between the two. My take is that unless you have a reason, use MPEG-Dash over HLS, as it’s newer and “smart” enough to adapt the video quality to match the viewer’s connection bandwidth.

So from now on in this article I’m going to be covering only MPEG-Dash examples. If you want to know how to watch HLS, check the repository documentation.

Watching the stream on VLC (Windows, Linux and MacOS)

If you have VLC installed, you can open it and press Ctrl + n or Cmd + n. On the network address input, enter:

https://192.168.1.192:4443/dash/cam1.mpd?watchkey=16356b9f

Replace 192.168.1.192 with your NGINX server address.

Hit enter, accept the certificate issue (it’s a self signed SSL certificate) and wait for the stream to begin:

VLC on Linux

Watching the stream on mobile (Android and iOS)

For that any MPEG-Dash (and HLS) compatible mobile client will work. I don’t have an iOS device here to test, but I’m sure there’s plenty of clients to choose. For Android I’m going to use the ExpressPlayer.

For some reason this app does not implement HTTPS (shame!), so I’m going to use HTTP in this example.

Open the app, select “CUSTOM INPUT”, then on “Media/MS3 URL …” add:

http://192.168.1.192:4080/dash/cam1.mpd?watchkey=16356b9f

Click “Play” and select “WV-DASH-M4F”:

Playing on Android

Watching the stream on the web (Google Chrome, Firefox, etc.)

There are two ways I know of for playing MPEG-Dash on browsers: Dash.js and Google’s Shaka Player, both JavaScript implementations.

Here I’m going to display Google’s Shaka in action.

Also, as we’re using a self signed certificate, Shaka will complain about not being able to verify the authority, so here too we’ll need to use HTTP instead of HTTPS. But this should be fixed if ever on production, as proper certificates will be used.

index.html:

<!DOCTYPE html>
<html>
  <head>
    <script src="js/shaka-player.compiled.js"></script>
    <script src="js/app.js"></script>
  </head>
  <body>
    <video id="video" width="1280" controls autoplay></video>
  </body>
</html>

app.js (taken from here)

const manifestUri =
      'http://192.168.1.192:4080/dash/bbb.mpd?watchkey=16356b9f';

function initApp() {
  shaka.polyfill.installAll();

  if (shaka.Player.isBrowserSupported()) {
    initPlayer();
  } else {
    console.error('Browser not supported.');
  }
}

async function initPlayer() {
  const video = document.getElementById('video');
  const player = new shaka.Player(video);

  window.player = player;
  player.addEventListener('error', onErrorEvent);

  try {
    await player.load(manifestUri);
  } catch (e) {
    onError(e);
  }
}

function onErrorEvent(event) {
  onError(event.detail);
}

function onError(error) {
  console.error('Error code', error.code, 'object', error);
}

document.addEventListener('DOMContentLoaded', initApp);

And it works like a charm, it’s the best player I’ve tested so far.

Thumbs up

Bonus: We don’t need NGINX for a single viewer

If your goal is to have a single computer consuming the video stream, eg.: RPi sends webcam video to a desktop computer to be recorded. Then you don’t need a NGINX server, you can stream directly from the RPi to the desktop computer using just FFmpeg and VLC over the much simpler RTP protocol.

Some comments on that:

  • You will need to decide between playback or record, I failed to perform both on VLC without crashing.
  • The record-and-play delay is a lot smaller, around 10 seconds on my setup.
  • No encryption nor authentication.

First, let’s name the machines:

RPi - 192.168.1.193
Desktop - 192.168.1.155

Start to stream the webcam video, from the RPi:

ffmpeg \
    -f video4linux2 -framerate 25 -video_size 1280x720 -i /dev/video0 \
    -f alsa -ac 2 -i sysdefault:CARD=WEBCAM \
    -c:v libx264 -b:v 1600k -preset ultrafast \
    -x264opts keyint=50 -g 25 -pix_fmt yuv420p \
    -c:a aac -b:a 128k \
    -vf "drawtext=fontfile=/usr/share/fonts/dejavu/DejaVuSans-Bold.ttf: \
text='CLOUD DETECTION CAMERA 01 UTC-3 %{localtime\:%Y-%m-%dT%T}': [email protected]: fontsize=16: x=10: y=10: box=1: boxcolor=black: boxborderw=6" \
    -f rtp_mpegts "rtp://192.168.1.155:5000?ttl=2"

Note that I’m pointing to the Desktop machine even though there’s no server running there. rtp://192.168.1.155:5000?ttl=2. I’ve also chosen the port 5000, but that’s up to you.

Now, on the Desktop computer open VLC and press Ctrl + n or Cmd + n to open the network stream dialog. And on the address input enter: rtp://192.168.1.155:5000. Then press Enter to start watching the stream.

That’s correct, we’re entering the same IP of the Desktop computer. That’s the computer which is receiving the stream.

That’s it

Video streaming is definitely fun. I can see many projects where this will be useful. I’m also feeling happy after learning how to set up these configurations and I hope you have enjoyed it as well. Feel free to contact me if you’re facing issues or just if you want to chat.

See you!

Sirius with 300mm focal length on a Canon EOS 600D

A wonderful night with essentially no clouds and no Moon. For a long time I’ve been planning to image Sirius A, the brightest star of our night sky. A main sequence star on Canis Major.

I’ve set up KStars on my notebook to wireless control my camera and mount using INDI on my homebuilt science station, so I programmed the set of exposures and left it running over the night while I enjoyed other targets with my dobsonian.

Author: Fernando Schuindt
License: CC BY 4.0
Camera: Canon EOS 600D
Lens/Telescope: Lens Zuiko Olympus 300mm f/4.5 Manual-Focus OM (Film)
Composition: 42x Light Frames 28” (19.6’ total) ISO 800 f/8, 25x Dark, 25x Flat, 25x Bias
Processing: Stacking with DeepSky Stacker, Photoshop Levels adjustments and Camera Raw Filter
Location Name: Aracaju - Sergipe, Brazil
Bortle Scale: 7
Location Aprox. Coordinates: 10°58’31.0”S 37°04’26.0”W
Timestamp (Local Time): From 11-23-2020 00:00:00 to 11-23-2020 00:33:10
Timezone: UTC-3 (No daylight saving time)
Outdoor temp.: 26° C
Outdoor humidity: 96%
Theme: Sirius
Center (RA, hms): 06h 45m 28.570s
Center (Dec, dms): -16° 44’ 31.236”
Legacy Surveys sky browser: Click here
Other Gear: Meade LX85 GEM unguided (Two-star alignment), OM-EOS adapter, homebuilt INDI wireless control station, KStars connected to INDI running on a Manjaro Linux x86 notebook. Astrometry: http://nova.astrometry.net/user_images/4171943#annotated
Backup Images (On Google Photos): https://photos.app.goo.gl/c2LqFysiHfPPEwbs7

(Click to zoom)

Picture

Annotated: Picture with notes

Equipment: Gear 1

Gear 2

Gear 3

How to use specific Ruby and Node.js legacy versions on Alpine with Dockerfile

While onboarding on a new project I ended up needing to build a Docker image for legacy versions of both Ruby and Node.js, more specifically Ruby 2.4.0 and Node.js 9.9.0. Unable to find good instructions on how to do so, I decided to write my own.

I saw two options here, one was to start from Ruby 2.4.0 official image and then install Node 9.9.0 on it; The other was the contrary, to start with Node and install Ruby. I’ve opted for the latter as I’m more familiar with the manual Ruby setup. So I’ll be starting from the image node:9.9.0-alpine.

I’ve looked both into RVM and asdf as version manager options for Ruby, really trying to ease out the Ruby installation process, but further inspection revealed that neither were built with containers in mind, it gets funky to try to set them up on a Alpine container. Not impossible, but funky. Manually compiling Ruby proved to be much easier to me, so I’m going this path.

So the steps would be:

  • Install system dependencies.
  • Use bash as the default shell (optional).
  • Download, compile and install Ruby 2.4.0 source code.

And here is it:

FROM node:9.9.0-alpine

ARG RUBY_RELEASE="https://cache.ruby-lang.org/pub/ruby/2.4/ruby-2.4.0.tar.gz"
ARG RUBY="ruby-2.4.0"

RUN apk add --no-cache git make gcc g++ libc-dev pkgconfig \
    libxml2-dev libxslt-dev postgresql-dev coreutils curl wget bash \
    gnupg tar linux-headers bison readline-dev readline zlib-dev \
    zlib yaml-dev autoconf ncurses-dev curl-dev apache2-dev \
    libx11-dev libffi-dev tcl-dev tk-dev

SHELL ["/bin/bash", "-l", "-c"]

RUN wget --no-verbose -O ruby.tar.gz ${RUBY_RELEASE} && \
    tar -xf ruby.tar.gz && \
    cd /${RUBY} && \
    ac_cv_func_isnan=yes ac_cv_func_isinf=yes \
    ./configure --disable-install-doc && \
    make && \
    make test && \
    make install

RUN cd / && \
    rm ruby.tar.gz && \
    rm -rf ${RUBY_NAME}

You can change the Ruby and Node.js versions to be the ones you need, I don’t think the process will dramatically change from version to version.

This code is on GitHub and DockerHub as zfschuindt/node_and_ruby:node-9.9.0_ruby-2.4.0.

C/2020 M3 ATLAS Comet with 300mm focal length on a DSLR

That’s my first stacked picture taken since my first polar alignment, I’m super excited!

Author: Fernando Schuindt
License: CC BY 4.0
Camera: Canon EOS 600D
Lens/Telescope: Lens Zuiko Olympus 300mm f/4.5 Manual-Focus OM (Film)
Composition: 19x Light Frame 40” (12.66’ total) ISO800 f/4.5, 25x Dark, 25x Flat, 25x Bias
Processing: Stacking with DeepSky Stacker, Photoshop Levels adjustments and Camera Raw Filter
Location Name: Aracaju - Sergipe, Brazil
Bortle Scale: 7
Location Aprox. Coordinates: 10°58’31.0”S 37°04’26.0”W
Timestamp (Local Time): From 11-13-2020 23:25:24 to 11-13-2020 23:54:30
Timezone: UTC-3 (No daylight saving time)
Theme: Comet C/2020 M3 ATLAS
Center (RA, hms): 05h 25m 58.341s
Center (Dec, dms): +03° 59’ 31.651”
Legacy Surveys sky browser: Click here
Other Gear: Meade LX85 GEM unguided (Two-star alignment), OM-EOS adapter, homebuilt INDI wireless control station, KStars connected to INDI running on a Manjaro Linux x86 notebook. Astrometry: https://nova.astrometry.net/user_images/4145606
Full Resolution Images (On Google Photos): https://photos.app.goo.gl/ePXtc8o39rtynfs28

Picture

Annotated: Picture with note

Equipment: Equipment 1

Equipment 2

Kugelblitz telescope

The Kugelblitz is a handmade Newtonian reflector telescope. This project started on November 19th, 2019 when I decided to assemble my own Newtonian. 10 months after I can say the project is ready for clear skies.

I’m publishing this page to serve as a documentation/specification for this instrument.

Specs.

  • Tube dimensions: 260x1455mm
  • Primary mirror diameter: 210mm (≈ 8.27”)
  • Focal length: 1390mm
  • Focal ratio: f/6.62
  • Distance from the secondary to the focal plane: 275mm
  • Visual secondary mirror dimensions: 60x80mm
  • AP secondary mirror dimensions: 70x98mm
  • OTA weight: 7.9kg
  • Dobsonian mount weight: 9.15kg

Staff & Off-the-shelf

  • Optic engineering, executive and assembly: Fernando Schuindt
  • Assembly date: Oct, 2020
  • First light date: Oct 17th, 2020
  • Assembly location: Aracaju - SE; Brazil
  • Primary mirror artisan: Sebastião Santhiago Filho
  • Primary mirror crafting date: Dec, 2019
  • Primary mirror crafting location: São Paulo - SP; Brazil
  • Visual secondary mirror artisan: Roberto Fornaciari (Pending)
  • Visual secondary mirror crafting date: Pending
  • Visual secondary mirror crafting location: São Paulo - SP; Brazil (Pending)
  • OTA artisan: Douglas Lucyrio (Telescópios Matão)
  • OTA crafting date: Sep, 2020
  • OTA crafting location: Matão - SP; Brazil
  • Dobsonian mount artisan: Douglas Lucyrio (Telescópios Matão)
  • Dobsonian mount crafting date: Sep, 2020
  • Dobsonian mount crafting location: Matão - SP; Brazil
  • Focuser manufacturer: Orion
  • AP secondary mirror manufacturer: GSO

Optical Project

Visual Setup

AP Setup

Diagonal off-axis illumination

Off-axis illumination

Results:
Smallest possible diagonal = 32.62 (mm)
Smallest user defined diagonal = 60 (mm)
Diagonal size maximizing illumination integrated across the field = 60 (mm)
Diagonal size for most even illumination across the field = 70 (mm)
illumination integrated across the field:  60: 90.8%; 70: 88.9%;

Central obstructions are 60: 28.6%; 70: 33.3%;
RMS wave deformations due to central obstruction are 60: 1/16; 70: 1/14;

Diagonal offset on fully illuminated field (towards primary mirror and away from focuser) = 1.84 (mm)
Diagonal offset (along diagonal face) = 2.61 (mm)
Diagonal offset on focal point (towards primary mirror and away from focuser) = 1.92 (mm)
Diagonal offset above focal point (towards primary mirror and away from focuser) (76.2 (mm) above) = 3.08 (mm)
Diagonal offset field edge (towards primary mirror and away from focuser) = 1.41 (mm)

Off-axis mask results:
maximum off-axis diameter = 76.84 (mm)
highest magnification = 76x
Dawes' Limit = 1.5 arc seconds
limiting magnitude = 13.2

Pictures

1

2

3

4

5

6

7

8

9

10

Special thanks to Catarina Dantas and the following Cloudy Nights members:

B 26354, SarverSkyGuy, Garyth64, JoeInMN, brebisson, GDAstrola, photomagica, Jon Isaacs, kathyastro, KLWalsh, dogbiscuit and Star Shooter.

Sunlight beam progression over a few weeks

This post was originally published on my old blog dedicated to amateur astronomy.

Recently I was working at my office when I noticed a light beam that wasn’t there on the previous days. I knew it was going to get bigger during the next days, so I decided to record it for a few weeks, plot a chart and make some calculations out of it, just for fun.

In total 14 days were recorded, but not in a 14-day time span. The first picture was taken August 20th and the last on September 12th. The missing pictures relates to days in which the weather was blocking the beam to be visible.

I used a fixed metric ruler on the wall and took a picture every day roughly at the same time (4:30pm).

1

It changed every day. Not only the beam size was getting bigger but the “furthest” and “nearest” point were  both moving to the left. This chart plots the evolution:

2

That’s the CSV data I created for this chart:

id,file,taken_at,starts,ends
1,P8204296.jpg,2020-08-20T16:29,0.1,3.1
2,P8214342.jpg,2020-08-21T16:33,2.4,6.2
3,P8224394.jpg,2020-08-22T16:37,4.3,9
4,P8234409.jpg,2020-08-23T16:33,4.8,9.4
5,P8244481.jpg,2020-08-24T16:33,5.9,11
6,P8254490.jpg,2020-08-25T16:31,6.7,12.1
7,P8264510.jpg,2020-08-26T16:33,8.4,14.5
8,P8274534.jpg,2020-08-27T16:34,9.6,16.2
9,P8284562.jpg,2020-08-28T16:33,10.8,17.8
10,P9034712.jpg,2020-09-03T16:33,18.5,28.5
11,P9044761.jpg,2020-09-04T16:33,19.9,30.4
12,P9064818.jpg,2020-09-06T16:31,22.2,33.3
13,P9094866.jpg,2020-09-09T16:34,26.8,40,0
14,P9124956.jpg,2020-09-12T16:34,30.9,42.9

And that’s the R script I wrote to plot it:

sunlight.data <- read.csv(file="~/sunlight_experiment.csv")
sunlight.data$taken_at <- as.POSIXct(sunlight.data$taken_at, format = "%Y-%m-%dT%H:%M", tz = "America/Maceio")
sunlight.data$size <- (sunlight.data$ends - sunlight.data$starts)

ggplot(sunlight.data, aes(x=taken_at)) +
  labs(title = "Sunlight beam projection over a few weeks") +
  geom_line(aes(y = starts, color = "darkred")) +
  geom_line(aes(y = ends, color = "darkblue")) +
  geom_line(aes(y = size, color = "darkgreen")) +
  scale_color_discrete(name = "Labels", labels = c("Furthest Point", "Beam Size", "Nearest Point")) +
  xlab("Date") +
  ylab("Line Point (cm)")

I also wrote a super simple Elixir program to compute the growth average for these values:

defmodule SunLightExperiment do
  @moduledoc false

  alias Decimal, as: D
  require Logger

  @doc false
  def perform do
    with data <- read_data("data.csv"),
         starts_growth <- compute_growth(data, :starts),
         ends_growth <- compute_growth(data, :ends),
         size_growth <- compute_growth(data, :size),
         starts_growth_average <- average(starts_growth),
         ends_growth_average <- average(ends_growth),
         size_growth_average <- average(size_growth) do
      Logger.info("Printing results...")

      IO.inspect(data)
      IO.puts("\n'starts' growth: #{inspect(starts_growth)}\n")
      IO.puts("'ends' growth: #{inspect(ends_growth)}\n")
      IO.puts("'size' growth: #{inspect(size_growth)}\n")
      IO.puts("'starts' growth average: #{inspect(starts_growth_average)}\n")
      IO.puts("'ends' growth average: #{inspect(ends_growth_average)}\n")
      IO.puts("'size' growth average: #{inspect(size_growth_average)}")
    end
  end

  @spec read_data(String.t()) :: Enumerable.t()
  defp read_data(file) do
    file
    |> File.stream!()
    |> CSV.decode()
    |> Stream.take(10)
    |> Stream.map(&get_valid_row/1)
    |> Stream.drop(1)
    |> Stream.map(&trim_columns/1)
    |> Stream.map(&to_entry/1)
    |> Enum.filter(fn entry -> entry != %{} end)
  end

  @spec compute_growth(list(map()), atom()) :: list(D.t())
  defp compute_growth(results, v) do
    Enum.reduce(results, [], fn result, acc ->
      case result == Enum.at(results, 0) do
        true ->
          acc

        _any ->
          acc ++ [D.sub(Map.get(result, v), tnm1(results, v, Enum.count(acc) + 1))]
      end
    end)
  end

  @spec average(list(D.t())) :: D.t()
  defp average(list) do
    with count <- Enum.count(list),
         count <- D.new(count),
         sum <- Enum.reduce(list, D.new(0), fn e, acc -> D.add(e, acc) end) do
      D.div(sum, count)
    end
  end

  @spec tnm1(list(map()), atom(), integer()) :: D.t()
  defp tnm1(results, variable, n) do
    results
    |> Enum.at(n - 1)
    |> Map.get(variable)
  end

  @spec get_valid_row({:ok, list(String.t())} | any()) :: list(String.t())
  defp get_valid_row(result) do
    case result do
      {:ok, row} -> row
      _any -> []
    end
  end

  @spec to_entry(list(String.t())) :: {:ok, Entry.t()} | {:error, Error.t()}
  defp to_entry([id, filename, taken_at, starts, ends]) do
    with {:ok, taken_at, _offset} <- DateTime.from_iso8601(taken_at),
         {starts, _any} <- D.parse(starts),
         {ends, _any} <- D.parse(ends),
         entry <- do_to_entry(id, filename, taken_at, starts, ends) do
      entry
    end
  end

  defp to_entry(_any) do
    %{}
  end

  @spec do_to_entry(String.t(), String.t(), DateTime.t(), D.t(), D.t()) :: map()
  defp do_to_entry(id, filename, taken_at, starts, ends) do
    %{
      id: id,
      filename: filename,
      taken_at: taken_at,
      starts: starts,
      ends: ends,
      size: D.sub(ends, starts)
    }
  end

  @spec trim_columns(list(String.t())) :: list(String.t())
  defp trim_columns(row) do
    Enum.map(row, fn
      column when is_binary(column) -> String.trim(column)
      column -> column
    end)
  end
end

According to Wakatime I took 1 hour and 33 minutes to write this one:

3

And the results are:

  • The “nearest point” moved to the left with a average speed of 1.34cm per day.
  • The “furthest point” moved to the left with a average speed of 1.84cm per day.
  • The “beam size” grew about 0.5cm per day.

I made the calculations using only the first 8 days, as they were separated with a almost precise 24h interval. This whole thing was a proof of concept for a later iteration of this experiment. The amount of data and the lack of precision yielded funky numbers, but this was expected in some sense. I think the overall outcome of this experiment is positive, I feel ready to start processing some more serious data.

And of course, that’s the beam on the first day, August 20th:

4

Now it on the last day, September 12th:

5

Click here to see all the images.

Mars occultation by the Moon video with Meade ETX 90 and Canon EOS 600D

This post was originally published on my old blog dedicated to amateur astronomy.

Author: Fernando Schuindt
License: CC BY 4.0
Camera: Canon EOS 600d
Lens/Telescope: Meade ETX 90 (OTA-only) (90mm aperture, 1250mm focal length, f/13.88, Maksutov-Cassegrain)
Location Name: Aracaju - Sergipe, Brazil
Location Aprox. Coordinates: 10°58’31.0”S 37°04’26.0”W
Other Gear: Meade LX85 GEM (unguided, no alignment), Canon EOS to M42 adapter, M42 to 1.25” adapter, Celestron Omni 2x Barlow
Processing: X and Y axis flip using kdenlive
Timestamp (Local Time): 06-09-2020 00:00:00 (aprox.)
Timestamp (UTC): 06-09-2020 03:00:00 (aprox.)

Video:

My first light with the Meade ETX 90, beautiful waning gibbous Moon

This post was originally published on my old blog dedicated to amateur astronomy.

Last week I bought an used Meade ETX 90 (90mm Maksutov), the fork mount was broken so I bought only the OTA to be used with my Meade LX85 equatorial mount. The OTA also had a problem with the 90° flip mirror mechanism to change the focuser image position, it was stuck on the upright 90° position, which is not only a bad place to fix a DSLR but it also makes the visual/AP change process a complicated one. I’ll manage a way to fix this mechanism ASAP, but in the meantime I’m using the DSLR in the upright position.

Another issue I had was my LX85 counterweight is way too heavy for this kind of payload. I had an old EQ1 mount with a smaller weight ideal for it, but the shaft role was smaller than the size of the LX85’s shaft. So I went to a local metal workshop (Acej) and asked them to enlarge the shaft hole for me, they couldn’t do it better, I got it as a perfect fit for the bigger shaft and then was able to balance the payload. Thanks to Andre Figueirêdo for helping me with subjects around 90mm Maksutovs and for giving me the Acej reference. He’s a really talented local astrophotographer.

That aside, I can say it’s superb equipment for planets and the Moon. Had an incredible night with Catarina gazing between Jupiter, Saturn and the Moon. I got really happy watching her faces and expressions while gazing through this scope, I think it’s a night I will never forget. The weather was also surprisingly comfortable.

She went to sleep around midnight, this time I was switching from visual to photography.

I also had brought my notebook to the garden and I was willing to use it to control the camera and the mount, but the Camera->Computer connection is a tricky one. 10m long USB cables are far from the ideal, but it works. Unless you’re passing through the USB device to a virtual machine. The problem is that my notebook runs Linux, and I haven’t yet got lucky with DSLR/mount controls on it. I always do it on my Win10 desktop, but it’s a desktop, I can do it only inside my office, not in the garden. So for that I had a virtual Win10 running on my Linux notebook. Controlling the mount from it works OK, but the DSLR will keep disconnecting every minute.

I’m yet to manage a solution for this,  but for the night I ended up not using the computer, just the DSLR itself and the mount handheld controller.

No fancy/cool techniques were used in the shots, just straight-forward single light frames and Photoshop editing, so definitely those equipment can do a lot better. Also mind that even larger magnifications can be achieved when using a 3x barlow lens, which is a plan for the future.

When I was about to wrap up for the night, my neighbor  appeared and we started talking about telescopes.  I soon invited him to come in and observe (keeping safe distance, as we both observe Covid-19 quarantine) and I think he enjoyed it a lot. He told me he owns a small refractor and that he’s studying astronomy, how cool? He said he’s starting his career on software development but programs since 2014, so we also spent a lot of time talking about IT.

Then I packed the stuff and got up back to the office, performed a really quick/crappy edition and called the night.

Author: Fernando Schuindt
License: CC BY 4.0
Camera: Canon EOS 600d
Lens/Telescope: Meade ETX 90 (OTA-only) (90mm aperture, 1250mm focal length, f/13.88, Maksutov-Cassegrain)
Location Name: Aracaju - Sergipe, Brazil
Location Aprox. Coordinates: 10°58’31.0”S 37°04’26.0”W
Other Gear: Meade LX85 GEM, Canon EOS to M42 adapter, M42 to 1.25” adapter.
Full Resolution Images: https://photos.app.goo.gl/GG4DzQkPTRJuuFJQ7

Picture 1 Composition: 1x Light Frames @ 1/160” ISO800 f/13.88
Processing: Photoshop resize, Camera Raw Filter
Timestamp (Local Time): 07-11-2020 01:19:51
Timestamp (UTC): 07-11-2020 04:19:51
Theme: Waning gibbous Moon 78.06%

Picture 2 Composition: 1x Light Frames @ 1/100” ISO3200 f/13.88
Processing: Photoshop resize, Camera Raw Filter
Timestamp (Local Time): 07-11-2020 01:11:47
Timestamp (UTC): 07-11-2020 04:11:47
Theme: Waning gibbous Moon 78.06%
Other gear: Celestron Omni 2x Barlow

Picture 3 Composition: 1x Light Frames @ 1/160” ISO3200 f/13.88
Processing: Photoshop resize, Camera Raw Filter
Timestamp (Local Time): 07-11-2020 01:13:18
Timestamp (UTC): 07-11-2020 04:13:18
Theme: Waning gibbous Moon 78.06%
Other gear: Celestron Omni 2x Barlow

Picture 4 Composition: 1x Light Frames @ 1/25” ISO3200 f/13.88
Processing: Photoshop resize, Camera Raw Filter, crop, manual background noise removal
Timestamp (Local Time): 07-11-2020 00:58:06
Timestamp (UTC): 07-11-2020 03:58:06
Theme: Bonus picture, Saturn
Other gear: Celestron Omni 2x Barlow

Gear: Gear 1

Gear 2

Improvised DSLR single shot of the Southern Cross (Crux)

This post was originally published on my old blog dedicated to amateur astronomy.

It’s Brazilian carnival once more, one of the biggest local holidays around here. It’s a time of parties and lots of people in the streets. For me it’s being a time of retirement and relaxation. And by that I mean being alone in my house spending the entire holiday on the internet, or with friends and family in some not crowded small city with some good nature, away from all the craziness that is the carnival. Don’t get me wrong, it’s fun, but taking the time to enjoy nature and relax is too. So I did.

The oldest friend I have (He’s son of my father’s best friend, so I know him long enough to don’t even remember when we first met) invited me to spend the carnival in his family’s estate, 1 hour driving from the town. A really nice, beautiful and comfortable place. Thank you Iago, for everything.

It was my first time visiting the place, so I asked him about the night sky, and he reported it was heavily polluted. So I decided not to bring my astronomy equipment, just my wife’s DSLR. (Just in case). I haven’t even brought any tripod. Nothing but the DSLR. And oh boy! I should have brought everything, not only the night sky was really nice (probably a Bortle Scale 5) but we spent the first night with very little clouds.

So I decided to improvise, got myself a pillow and placed it on the ground with the DSLR on top pointing upwards and managed to take some shots. After some in-house picking of the best shot and some edition, here’s the result.

I really liked the fact that in this picture we can see not only the Southern Cross but also the Wishing Well Cluster (NGC 3532) and the Homunculus Nebula.

Author: Fernando Schuindt
License: CC BY 4.0
Camera: Canon EOS 600D
Lens/Telescope: Canon EF 40mm 1:2.8 STM Lens
Composition: 1x Light Frame 10” ISO400 f/2.8
Processing: Photoshop Levels adjustments and Camera Raw Filter
Location Name: Salgado (Povoado Turma) - Sergipe, Brazil
Location Aprox. Coordinates: 11° 02’ 11.2” S 37° 28’ 54.2” W
Timestamp (Local Time): 02-24-2020 01:43:12
Theme: Southern Cross (Crux)
Other Gear: None
Astrometry: http://nova.astrometry.net/user_images/3414424
Full Resolution Images: https://photos.app.goo.gl/anvQzjiFkiXSafXC9

Picture

Astrometry

Here’s a bonus picture of M42 with a coconut tree:

Bonus Picture

And some pictures of the estate:

Estate Picture 2

Estate Picture 3

Estate Picture 1

Again, thank you Iago for such great experience, and thank you Catarina, for once more allowing me to play with the camera. I love you honey!

Low budget imaging of NGC 3628 and NGC 3627

This post was originally published on my old blog dedicated to amateur astronomy.

This one took me quite a good time to process, ended up making 3 versions of it and sticking with the latest. It’s the first time I programmed a galaxy themed picture so my inexperience spoke during the whole process.

The plan was to put two or more galaxies in the frame with a low budget plan (the remaining usage time I had on iTelescope). I was aiming to the Leo Triplet, a group of 3 galaxies also known as the M66 group. But I was using a deep field telescope (0.43-m f/6.8 reflector + CCD + f/4.5 focal reducer) so I ended up taking just two of the three galaxies: NGC 3628 (the Hamburger Galaxy / M65 and NGC 3627 (the M66).

The image composition is the following:

  • 4x 30” Luminous frame
  • 4x 30” Red frame
  • 4x 30” Green frame
  • 4x 30” Blue frame

The images were stacked using DeepSky Stacker and stretched using Adobe Photoshop.

Author: Fernando Schuindt
License: CC BY 4.0
Camera: FLI-PL6303E CCD camera
Lens/Telescope: Planewave 17” CDK (431mm), Focal Length 1940mm (0.66 Focal Reducer), f/4.5
Composition: 4x Luminous Frame 30”, 4x Red Frame 30”, 4x Green Frame 30”, 4x Blue Frame 30”
Processing: DeepSky Stacker image stacking, Photoshop Channels composition, Levels adjustments, Image Mode and Camera Raw Filter
Location Name: New Mexico Skies Observatory (iTelescope), Mayhill - New Mexico, US.
Location Aprox. Coordinates: 32°54’14” N 105°31’44” W
Timestamp: 01-07-2020 11:22:57 UTC;
Theme: NGC 3628 (the Hamburger galaxy) and NGC 3627 (the M66)
RA: 11 20 15.10
DEC: +12 59 24.0
Other Gear: https://support.itelescope.net/support/solutions/articles/231906-telescope-21
Full Resolution Images: https://photos.app.goo.gl/EZGQK49ZGu4b2h719
Picture Information: https://gist.github.com/fschuindt/41f8683fad81136dfd859efff534a1c2
Astrometry: http://nova.astrometry.net/user_images/3312354#annotated

Picture

Astrometry

And this time a courtesy negative gray scale picture. Using the four luminous frames stacked:

Negative

My first picture at the SSO (NGC 3372)

This post was originally published on my old blog dedicated to amateur astronomy.

As I’m studying a lot about AP, I thought would be nice to give a try to iTelescope.net systems, and it was pretty fun. I picked the Siding Spring Observatory in Australia and the T8 telescope, which is a 106mm Takahashi FSQ Apochromatic, 0.10-m f/5 telescope. The target was NGC 3372, the Eta Carinae Nebula.

This is a quite special DSO to me, as it’s the first object that I got in love with and started my fascination for deep sky in general. It all started with a video by my favorite YouTube channel: Sixty Symbols. The video was this one here, with the professor Mike Merrifield.

I never had access to such equipment before, so I got really exited with the filter selection and ordered a 32” exposure for each filter available. I just wanted to see it, haha. So I ended up with some frames that I had no use for.

I used:

  • 1x 32” Luminous frame
  • 1x 32” Red filter frame
  • 1x 32” Green filter frame
  • 1x 32” Blue filter frame
  • 1x 32” Hydrogen Alpha filter frame

Just one short frame for each filter I wanted to use, because that’s a budget project, hahaha.

When the frames arrived I thought I had messed up with something, they looked really dark. But after merging the Red and Ha frames, then the other channels (Green and Blue) and composing the HaRGB image with some stretching in Adobe Photoshop, it looked really nice for my newbie taste. As the CCD on the camera is Black&White, the colors were composed by using R, G and B filters. In this case the Red channel is merged with the Hydrogen Alpha channel, to reveal Hydrogen based structures with more detail.

To give an idea, this is how one of the frames looks like without any processing (the Red frame, smaller size tho):

Red Channel

But after the processing, that’s the result:

Author: Fernando Schuindt
License: CC BY 4.0
Camera: CCD FLI Microline 16803
Lens/Telescope: 106mm Takahashi FSQ Apochromatic, 0.10-m f/5
Composition: 1x Luminous Frame 32”, 1x Red Frame 32”, 1x Green Frame 32”, 1x Blue Frame 32”, 1x Ha Frame 32”
Processing: AvisFV 2.0 format conversion, Photoshop Channels composition, Levels adjustments and Camera Raw Filter
Location Name: Siding Spring Observatory (iTelescope), New South Wales - Australia;
Location Aprox. Coordinates: 31° 16’ 14.52’’ S 149° 3’ 31.32’’ E
Timestamp: 01-04-2020 17:49:24 UTC;
Theme: NGC 3372 (Eta Carinae nebulae)
RA: 10 45 06.00
DEC: -59 52 00.0
Other Gear: https://support.itelescope.net/support/solutions/articles/231911-telescope-8
Full Resolution Images: https://photos.app.goo.gl/ULajFc5dyPzvZKqk9
HaRGB Picture Information: https://gist.github.com/fschuindt/996ef63e6cda908f390f8d7c8407b17e
RGB Picture Information: https://gist.github.com/fschuindt/e66a200e3f1e766b46b2f2f4ac66d1bf

Picture HaRGB

I’ve also produced another image without the Ha filter:

Picture RGB

Fun fact is that the NGC 3532 (Wishing Well Cluster) is right there in the bottom right corner, I loved it. So I’m already waiting for my new schedule to happen, I won’t reveal it yet, but I’m hopping to see two galaxies in the next picture.

Again a special thanks to Terry F. for his teachings.

Linkerd service mesh CPU footprint on Kubernetes

I’ve been using Linkerd for a while on staging environments, but now I finally deployed it to production. And I went to inspect its CPU/memory footprint on my older Prometheus/Grafana setup. The Kubernetes cluster for production is 3-node-sized with 2GB RAM each and it’s running at DigitalOcean. After a whole day since the initial setup (which is really simple) the result is what follows.

CPU at node 1/3 (with annotations)

Linkerd CPU footprint graph, node 1/3 with annotations

CPU at node 1/3

Linkerd CPU footprint graph, node 1/3

CPU at node 2/3

Linkerd CPU footprint graph, node 2/3

CPU at node 3/3

Linkerd CPU footprint graph, node 2/3

My reports on memory usage

This Kubernetes cluster is basically running a dozen BEAM instances in production (we’re running a few Elixir APIs). That makes my reports on memory kinda useless due to the way that the BEAM manages its memory. At the end of the day, the memory usage of my containers is tightly related with how long they’re running.

But if you want to see them anyway, here they are:

Setting up a residential K3s single node cluster with KVM in a Manjaro host and using Cloudflare dynamic DNS

press

For long I’ve been playing with Kubernetes in production environments, at my job and in other projects. But everything was done using managed solutions like the one from Digital Ocean, which is great by the way. But I was willing to put a bit of my hands on it.

I happen to have a desktop computer (i5 8400 16GB RAM running Manjaro) which I’m not using that much, so I planned to setup a VM on it and run a single node Kubernetes cluster myself. But wait a second, for that we had Minikube, right? Yes, the problem being it’s designed to work inside your own computer and not to have contact with the external world, not even your local network. I was having trouble to make that work when I remembered about K3s, which is a Kubernetes distribution that’s actually simpler than Minikube, it’s great and it’s also production ready.

One of the key things here is that I wanted to take advantage of Intel’s Vt-d and KVM for virtualization. Also I want this VM to get its own IP address on my local network, so to my router it would appear as another physical machine. For that I’m going to create a bridge network interface between the host and the VM.

Before we start, mind that all the work here will be made in my Manjaro laptop, named skywitch a.k.a. “laptop”, connecting via SSH to the Manjaro bare metal desktop server, redwitch a.k.a “the host” (The SSH setup was already in place). I want to be able to connect to the K3s from skywitch and also to access its running services from the outside world. Which will need some port-forwarding at the internet router firmware. (K3s already comes with Traefik as an Ingress resource)

For the VM OS I’m going to choose Ubuntu Server 18.04 and its hostname will be named warlock.

All the servers, redwitch and warlock will have static IPv4 address on the local network. All other devices, including my laptop will be using DHCP.

Here’s how it’s going to look like after finished:
the big picture

You can ignore the HDMI KVM Switch device for the purpose of this guide.

And just to make things clear, here’s the /etc/hosts I’ve setup to my skywitch laptop:

127.0.0.1	localhost
127.0.1.1	skywitch
192.168.1.151	redwitch
192.168.1.161	warlock

::1	localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Another important thing is that here in Brazil is common for residential connections to lack a static external IP address, but to have a dynamic one that changes each restart. In my case my address changes depending on the route to the server, so I can be talking to Spotify as one address and to YouTube as other, you never know, but they all point to me. Plus the ports 80 and 443 are blocked to the external world, so people can not serve web pages without having to explicitly use a port like :8080 in the URL.

And that means two things:

  1. I’m going to need some sort of Dynamic DNS tool (remember no-ip?), in this case Cloudflare will be the choice, as it provides this service for private domain names with no cost.

  2. The router port-forwarding step must account for having the ports 80 and 443 blocked.

I’m going to assume you already have a server with a KVM setup and Vt-d enabled. So I’m not going to cover this process here, this can be easily found on Google. But if you’re wondering, here is one of those guides.

So the plan is:

  • Setup a bridge network interface on the host.
  • Create a Ubuntu 18.04 VM on the host with the bridged network.
  • Set up a K3s single node Kubernetes cluster into the VM.
  • Describing a Dynamic DNS cronJob into the cluster to update a domain name in Cloudflare with the external IP address.
  • Configuring the router to forward certain ports to the VM.
  • Deploying a simple application to the cluster and have it exposed to the world.

So let’s start.

Just a quick note: Many of the stuff presented here was taken from other guides on the web, which you can find all listed at the end of this post.

Setup a bridge network interface on the host

This will allow us to create a VM that will receive its own IP on our local network, pretty much as every other physical device does.

First install bridge-utils, it will come in handy later:

$ sudo pacman -S bridge-utils

Now you need to find out the name of your main network interface, which can be done with $ ip link show or $ ifconfig. In my case it was named enp0s31f6.

You’re also going to need to know the connection gateway and DNS addresses.

To get the gateway address you can just:

$ route -n | grep "^0.0.0.0" | tr -s " " | cut -f2 -d" "

192.168.1.1

And to know the DNS:

$ cat /etc/resolv.conf

# Generated by NetworkManager
search GREATEK
nameserver 192.168.1.1

In my case they’re the same, but it may differ from vendor to vendor, I’m not sure.

You should also pick up a static IP address for the host machine, I’m going to choose 192.168.1.151. Plus I’m going to name the bridge interface as br1.

Now let’s create a /etc/netctl/kvm-bridge file:

$ sudo vi /etc/netctl/kvm-bridge

With the following content:

Description="Bridge Interface br10 : enp0s31f6"
Interface=br1
Connection=bridge
BindsToInterfaces=(enp0s31f6)
IP=static
Address='192.168.1.151/24'
Gateway='192.168.1.1'
DNS='192.168.1.1'
MACAddressOf=enp0s31f6
SkipForwardingDelay=yes
set IP=no

Replace the values with the ones you want, then:

$ sudo systemctl restart NetworkManager.service

Then start and enable the kvm-bridge:

$ sudo netctl start kvm-bridge
$ sudo netctl enable kvm-bridge

And that’s it. You can use $ brctl show and $ bridge link to check bridges and see bridged interfaces respectively. But for now I think we’re good. Our host now have a new source for its static IP (192.168.1.151) and a bridge network interface on it. Great.

Create a Ubuntu 18.04 VM on the host with the bridged network

I won’t get much into the libvirt usage, if you want here’s the virsh CLI reference page:
https://libvirt.org/sources/virshcmdref/html/

And the basics:

Boot a VM - virsh start <vm>
Stop a VM - virsh shutdown <vm>
Suspend a VM - virsh suspend <vm>
Delete a VM - virsh destroy <vm> and virsh undefine <vm>

We’re going to use a Shell script in order to create and configure the VM. So create a create_vm.sh file:

$ vi ~/create_vm.sh

With:

#!/bin/sh

if [ -z "$1" ] ;
then
 echo Specify a virtual-machine name.
 exit 1
fi

sudo virt-install \
     --name $1 \
     --ram 4096 \
     --disk path=/home/fschuindt/hdd_repo/libvirt/images/$1.img,size=30 \
     --vcpus 4 \
     --os-type linux \
     --os-variant ubuntu18.04 \
     --network bridge:br1,model=virtio \
     --graphics none \
     --console pty,target_type=serial \
     --location 'http://archive.ubuntu.com/ubuntu/dists/bionic-updates/main/installer-amd64/' \
     --extra-args 'console=ttyS0,115200n8 serial'

Here you shall stop and perform some editing in the file. A few things to look after are the RAM size, the number of CPUs and the --disk path, which mine is pointing to /home/fschuindt/hdd_repo/libvirt/images/ with a 30GB sized disk.

This directory on my host is on a 2TB HDD mount, it’s important you have it mapped to a physical device or to a place where you have enough space to install the VM. I just allocated 30GB, but that’s arbitrary. Just mind that this space will be actually occupied during the VM creation.

Also the --network bridge:br1,model=virtio part shall point to the br1 bridge interface we’ve created earlier in the guide.

Save and:

$ chmod +x create_vm.sh

$ ./create_vm.sh warlock

Where warlock is the name of the VM.

Now the S.O. installation will begin, the Ubuntu setup is pretty straight forward so you must complete it with no problem. Just mind tho, at the “Software selection” phase to pick up the OpenSSH server.

After finished the installation, it will reboot and end the process.

Let’s connect via SSH to the freshly installed VM, for that, find the VM IP using:

sudo nmap -sP 192.168.1.0/24

It will have the port 22 (OpenSSH) opened. In my case it was 192.168.1.12.

So, from my laptop:

Edit with your username (fschuindt for me) and it will ask you for the password you’ve set during installation. A good thing to do now is to set a static IP to the VM.

Edit the file /etc/netplan/01-netcfg.yml with:

# warlock VM

# This file describes the network interfaces available on your system
# For more information, see netplan(5).
network:
  version: 2
  renderer: networkd
  ethernets:
    enp1s0:
     dhcp4: no
     addresses: [192.168.1.161/24]
     gateway4: 192.168.1.1
     nameservers:
       addresses: [1.1.1.1,1.0.0.1]

I’m using my local values for gateway, I’m setting Cloudflare DNS as my DNS servers (1.1.1.1 and 1.0.0.1) and the VM local static IP to 192.168.1.161.

To apply just:

$ sudo netplan apply

You will get disconnected from the SSH session, but that’s ok, just connect again with the new IP. Better yet, add it to your /etc/hosts file with your VM name, as I showed in the beginning.

Another important thing to do is to upload your SSH public key to the VM and to disable password logins. I won’t be covering it here, for that just go to:
https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys-on-ubuntu-1804

Now we have a virtual Ubuntu 18.04 running with a static IP on our local network. Let’s do some work in it.

Set up a K3s single node Kubernetes cluster into the VM

There’s nothing much to say in this section really, K3s installation and usage is really simple, I recommend checking their homepage and the docs for a quick start. But here’s the basic:

This is inside the VM.

To install:

$ curl -sfL https://get.k3s.io | sh -

This will also configure systemd, so K3s will start after every reboot.

To get the admin .yaml file:

$ sudo cat /etc/rancher/k3s/k3s.yaml

I’m going to save this on my laptop’s ~/.kube/. Saving it named as config connects your kubectl with the new cluster.

Now our single node K3s cluster is up and running, you can already start playing with kubectl. The next steps are really for exposing it to the world, so if you don’t want that you can declare work done =]. Otherwise we still have some more stuff to do.

Describing a Dynamic DNS cronJob into the cluster to update a domain name in Cloudflare with the external IP address

So I plan to expose services running on the K3s cluster to the outside world. If I had a static external IP address that would be great, but as you may already know it’s not the case. But it’s still possible to have a domain name pointed to the server as a Dynamic DNS (DDNS) using one DDNS provider like no-ip.com and DynDNS. That requires a DDNS client running on my system checking changes on the external IP and updating it against the provider.

The provider will be Cloudflare, as it’s not only a DNS provider but also supports DDNS. I own the 722.network domain name on Cloudflare and I’m going to use the subdomain fschuindt.722.network to point to the cluster.

The client will be ddclient, a well known DDNS client written in Perl. I’ve setup a ddclient public Docker image that you can configure and use for that same matter. For using it I’m going to set up a Kubernetes cronJob that every 5 minutes will spawn a container using that image and run a DDNS check/update command, then exits and waits to the next execution, and so on.

It’s not perfect but it’s enough, that shall keep the subdomain name updated.

So if you will, create a ddclient-job.yml file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ddclient-config-map
  labels:
    owner: ddclient
data:
  LOGIN: "[email protected]"
  PASSWORD: "your-cloudflare-global-api-key"
  ZONE_DOMAIN: "your-domain.com"
  ZONE_HOSTNAME_1: "your-host.your-domain.com"
  ZONE_HOSTNAME_2: "other-host.your-domain.com"
  ZONE_HOSTNAME_3: ""
  ZONE_HOSTNAME_4: ""
  ZONE_HOSTNAME_5: ""
  ZONE_HOSTNAME_6: ""
  ZONE_HOSTNAME_7: ""
  ZONE_HOSTNAME_8: ""
  ZONE_HOSTNAME_9: ""
  ZONE_HOSTNAME_10: ""
---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: ddclient-job
  labels:
    owner: ddclient
spec:
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 5
  successfulJobsHistoryLimit: 5
  startingDeadlineSeconds: 60
  # At every 5th minute.
  schedule: "*/5 * * * *"
  jobTemplate:
    metadata:
      name: ddclient-job
      labels:
        owner: ddclient
    spec:
      activeDeadlineSeconds: 240
      backoffLimit: 3
      template:
        metadata:
          name: ddclient-job-pod
          labels:
            owner: ddclient
        spec:
          containers:
          - name: ddclient-job-container
            image: zfschuindt/ddclient:latest
            command: ["bash", "-c", "/ddclient/entrypoint.sh"]
            envFrom:
              - configMapRef:
                  name: ddclient-config-map
          restartPolicy: OnFailure

And edit it to fit your needs. Especially the ConfigMap section, where you want to provide your Cloudflare credentials with the domain/subdomain names. I’ve created space for up to 10 subdomains, but if you want more you can easily edit the entrypoint.sh file of the image.

Here’s the ddclient image GitHub repository if you want:
https://github.com/fschuindt/ddclient

Now we can just spawn the job into K3s:

$ kubectl apply -f ddclient-job.yml

Now someone is working to keep my external IP updated on the fschuindt.722.network domain name. Wonderful!

Configuring the router to forward certain ports to the VM

This part changes for everyone. It’s really dependent on which internet router vendor/model you have, but in general the concept is the same: Let the router to know which static IP address on the local network is to forward incoming connections on given ports.

If skipped, the outside world won’t be able to connect to the services on the cluster, as the router won’t know what to do with those connections, it must deliver it to some device on the network, but without knowing which it drops it.

For configuring this you need to access your router firmware interface, for me it’s on http://192.168.1.1/ for any wire-connected device on the network. Then you must provide credentials and look for any “port forward” option.

You can find the default user/password combination for your router as well as instructions for port forwarding (if it supports) on the PDF manual for your router model (every model has one, just check online).

For me I added two entries on the port-forwarding rules list. One forwarding every external connection on the port 7222 to the port 80 on the 192.168.1.161 (the warlock VM) and other forwarding every external connection on the port 7223 to the port 443 on the same server, the 192.168.1.161 (warlock VM).

It’s looking like this:

port-forwading

And that will let the router know to which device to send the incoming connections. One more thing to do, let’s deploy a service to the K3s and test the whole thing.

Deploying a simple application to the cluster and have it exposed to the world

This will be done at the skywitch laptop, connected to cluster using kubectl.

Right now we have only one DNS pointing to the cluster, which is fschuindt.722.network. I’m going to deploy a service to operate on this address, more precisely a Ingress resource.

The service we’re going to deploy is a simple HTTP “ping/pong” echo. It serves only one route GET /ping, which will reply 200 OK, "pong". I wrote this service using Elixir and it’s on GitHub here:
https://github.com/fschuindt/http_echo

It already comes with its own Dockerfile and its image is publicly available at DockerHub. Plus if you check the /k8s folder on the repository you will find a group of Kubernetes resources for deploying it into Kubernetes. This will make everything easier.

The resources are:

  • config_map.yml
  • service.yml
  • ingress.yml
  • deployment.yml

Important: The files in the repository are just examples and may differ a bit from the ones presented here.

And we’re going to create them in this order, so the ConfigMap first:

config_map.yml

apiVersion: v1
kind: ConfigMap
metadata:
  name: echo-config-map
  namespace: default
  labels:
    owner: echo
data:
  MIX_ENV: "prod"
$ kubectl create -f config_map.yml

Then the Service:

service.yml

apiVersion: v1
kind: Service
metadata:
  name: echo-service
  labels:
    app: echo
    owner: echo
spec:
  type: NodePort
  selector:
    app: echo
    tier: web
  ports:
  - port: 4080
    targetPort: 4080
$ kubectl create -f config_map.yml

The Ingress:

ingress.yml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: echo-ingress
spec:
  rules:
  - host: echo.warlock.network
    http:
      paths:
      - backend:
          serviceName: echo-service
          servicePort: 4080
  - host: fschuindt.722.network
    http:
      paths:
      - backend:
          serviceName: echo-service
          servicePort: 4080

Notice here I’m also setting the echo.warlock.network domain name to be used within the local network, I’m going to add this hostname on my laptop’s /etc/hosts as well.

$ kubectl apply -f ingress

And finally the deployment:

deployment.yml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
 name: echo-deployment
 namespace: default
 labels:
    owner: echo
    app: echo
    tier: web
spec:
  revisionHistoryLimit: 5
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        owner: echo
        app: echo
        tier: web
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: echo-container
          image: zfschuindt/http_echo:latest
          imagePullPolicy: Always
          ports:
            - containerPort: 4080
          args: ["bash", "-c", "./app/entrypoint.sh"]
          envFrom:
            - configMapRef:
                name: echo-config-map
          livenessProbe:
            httpGet:
              path: /ping
              port: 4080
$ kubectl create -f deployment.yml

And that shall make the http://fschuindt.722.network:72222/ping available and serving the HTTP echo service to the world. :)

It may be already offline by the time you’re reading this, but believe me, it worked.

And that’s how my Kubernetes Dashboard looks like:

kubernetes dashboard

By the way, if you want to install this dashboard on your cluster, check how to do it here.

And with that we shall have our residential K3s cluster running and serving to the outside world on top of a fast and optimized virtualization method. I hope this guide have served you well, setting up this environment brought me new ideas for residential server setups and was a lot of fun!

Thank you for staying with me.

See you soon. :)

References

Cover picture: “Gutenberg Publishes the World’s First Printed Book (Illustration) Civil Rights Medieval Times Famous Historical Events Visual Arts”

Concurrent Calculation Of Fibonacci In Elixir

There’s a known formula to evaluate a n Fibonacci term position without iteration or recursion. Formula

This formula give us a precise approximation:

  • Term n(26) = 121393.00000000009
  • Term n(1200) = 2.7269884455407366e250

Which according with this list, is fine:

  • Term n(26) = 121393
  • Term n(1200) = 272.698844554059974143456202000 × 10²⁷

The actual 1200th term position is 251 digit long, so I’ve posted a equivalent notation here.

A cool (and unrelated) fact is that it uses the Golden Ratio in the calculation (1 + √5) / 2, and what seems a reverse form of it (1 - √5) / 2.

So, the formula I’ve described as:

defmodule FibonacciCalculus do
  @golden_n :math.sqrt(5)

  def of(n) do
    (x_of(n) - y_of(n)) / @golden_n
  end

  def x_of(n) do
    :math.pow((1 + @golden_n) / 2, n)
  end

  def y_of(n) do
    :math.pow((1 - @golden_n) / 2, n)
  end
end

By the way, if you haven’t already wondered, after the term 1474 it crashes:
Crash output

That’s because Erlang’s math module operates only with numbers that can be represented with floating points, and such a big number can’t.

But anyway, my goal here isn’t to calculate Fibonacci at all.
I’m just wanting to see Elixir’s power doing hundreds of those big number calculations at the same time.

I want to give Elixir a sequential integer list, in this case from 1 to 1474, which is the apparent limit. Then ask it to spawn a Erlang process for each one of those elements. Each process should receive the element and return the result of its FibonacciCalculus.of(n), being n the element.

I should end up with a unordered result, as reflect of the concurrent computation.

So, I’ve described the module:

defmodule ConcurrentFibonacci do
  def start do
    concurrent_map(1..1474, fn(x) ->
      "Fibonacci of #{x} is: #{FibonacciCalculus.of(x)}"
    end)
  end

  def concurrent_map(list, func) do
    list |> Enum.map(fn e -> spawn(fn ->
      IO.puts func.(e)
    end) end)
  end
end

And the result is beautiful:
Demonstration 1

A simple code and it executes more than 1400 concurrent processes in less than a blink of an eye.

Also the unordered result is evident:

[...]
Fibonacci of 1456 is: 8.640108610267577e303
Fibonacci of 1455 is: 5.3398807876359814e303
Fibonacci of 1457 is: 1.397998939790356e304
Fibonacci of 1458 is: 2.262009800817114e304
Fibonacci of 1460 is: 5.922018541424584e304
Fibonacci of 1465 is: 6.567619203443404e305
Fibonacci of 1470 is: 7.28360130920199e306

In an attempt to go deeper, instead of using the 1..1474 list, I’m using the following function:

def list do
  Enum.take(Stream.cycle(1..1474), 10_000)
end

This will repeat the 1..1474 list approximately 7x, to result in a list containing 10.000 elements.

Let’s see how it goes:
Demonstration 2

You can see it takes more time, obviously.
But let’s think about it. It’s ten thousand concurrent executions, ten thousand processes.

I have even tested with 100.000 processes, in my 4GB RAM [email protected] notebook it takes 5.47 seconds. One hundred thousand processes in 5 seconds.

I have to say I’m impressed.

I’m about one year playing with Elixir in my spare time, it’s a really fun language, especially if you never programmed functional before.

Ruby Gem To Verify Firebase Id Token Signatures

Article screenshot

This is a off blog’s topic, but I just released a new Ruby gem.

The gem firebase_id_token was developed to easily verify Firebase ID Token signatures in Ruby back-end environments. It uses Redis to store Google’s x509 certificates, which helps other processes in your application to access it really fast.

The Firebase ID Token is really a JWT. What the gem does is to check if the token was made for your application and if it’s valid, both in it’s parameters and in it’s RSA signature.

Check out the gem’s Github for more info. :)

Structure And Interpretation Of Computer Programs

Book cover illustration

Also referred as SICP, it’s a well known MIT book published in the late 80’s and used in the Computer Science and Electrical Engineering courses. Written by Harold Abelson, Gerald Jay Sussman and Julie Sussman. It’s the definitive functional programming book.

It’s available for free in many formats such as HTML and PDF. This book insipred me to create this blog. It’s just wonderful, it’s the great grimoire of computer wizardry. But not only that, the MIT OpenCourseWare has the whole video lectures from 1986 taught by Harold Abelson and Gerald Jay Sussman themselves, and with no cost.

These twenty video lectures by Hal Abelson and Gerald Jay Sussman are a complete presentation of the course, given in July 1986 for Hewlett-Packard employees, and professionally produced by Hewlett-Packard Television. These videos are also available here under a Creative Commons license compatible with commercial use.

Note: These lectures follow the first edition (1985) of Structure and Interpretation of Computer Programs. Many of the programs discussed were rewritten for the second edition (1996) of the book, and new material was added. These video lectures will still be useful for students using the second edition, since the overall themes of the course and order of presentation are unchanged.

These videos are courtesy of Hal Abelson and Gerald Jay Sussman, and are used with permission.

The book uses the language Scheme which is a Lisp dialect, one of the oldest languages (1959) to teach pure functional concepts from the very basics to the more complex cases. It’s like learning to program again from a pure functional perspective.

This course introduces students to the principles of computation. Upon completion of 6.001, students should be able to explain and apply the basic methods from programming languages to analyze computational systems, and to generate computational solutions to abstract problems. Substantial weekly programming assignments are an integral part of the course. This course is worth 4 Engineering Design Points.

And here’s the first lecture, just in case:

Course Citation and License

Eric Grimson, Peter Szolovits, and Trevor Darrell. 6.001 Structure and Interpretation of Computer Programs. Spring 2005. Massachusetts Institute of Technology: MIT OpenCourseWare, https://ocw.mit.edu. License: Creative Commons BY-NC-SA.

My Elixir Study Notes

Article illustration

While studying the Elixir Getting Started Guide I decided to begin resuming it out as a way to get it fixed. I ended up with a very useful document for those who already read the guide and want to quickly review any topic. It’s a one page document resuming the whole guide, which is just great to use with the Find command.

I basically copied, slightly altered and omitted great part of the content to focus on things I thought more important (to me). But provides a quick explanation of everything. Also includes some notes I made.

As this post still serve as a study resource, it’s constantly updated.



Notes


  • Get some help:
iex> i 'hello'
Term
  'hello'
Data type
  List
Description
  ...
Raw representation
  [104, 101, 108, 108, 111]
Reference modules
  List
  • When “counting” the number of elements in a data structure, Elixir also abides by a simple rule: the function is named size if the operation is in constant time (i.e. the value is pre-calculated) or length if the operation is linear (i.e. calculating the length gets slower as the input grows).
  • String concatenation is done with: <>.
  • Operators or, and, not can only accept boolean values. Besides these boolean operators, Elixir also provides ||, && and ! which accept arguments of any type. For these operators, all values except false and nil will evaluate to true.
  • The variable _ is special in that it can never be read from. Trying to read from it gives an unbound variable error.
  • Guard Clauses are neat:
# Anonymous functions can have guard clauses:
# They also apply to the 'case' statement, 'when'.
iex> f = fn
...>   x, y when x > 0 -> x + y
...>   x, y -> x * y
...> end
#Function<12.71889879/2 in :erl_eval.expr/5>
iex> f.(1, 3)
4
iex> f.(-1, 3)
-3

Basic Types


Booleans

All good, just true and false. Nothing special.

Atoms

Pretty much like Lisp’s Atoms, A.K.A. Symbols in Ruby.

Anonymous Functions

Anonymous Functions (function literal, lambda abstraction) is delimited between fn and end.

# first class citizens (can be passed as arguments)
iex> add = fn a, b -> a + b end
iex> add.(3, 2)

Anonymous functions are closures and as such they can access variables that are in scope when the function is defined.

Lists

Describes itself.

# Add or subtract using ++ or --
iex> [2, 23, 42, 11, true]
iex> list = [1, 2, 3]
# Get head and tail.
iex> hd(list)
1
iex>tl(list)
[2, 3]
  • When Elixir sees a list of printable ASCII numbers, Elixir will print that as a char list (literally a list of characters).
  • Single-quotes are char lists, double-quotes are strings.

Tuples

Similar to lists, but stored in memory, all data is availible with no recursion needed.

iex> {:ok, "hello"}
{:ok, "hello"}
iex> tuple_size {:ok, "hello"}
2

List or Tuples?

Accessing the length of a list is a linear operation: we need to traverse the whole list in order to figure out its size. Updating a list is fast as long as we are prepending elements.

Tuples, on the other hand, are stored contiguously in memory. This means getting the tuple size or accessing an element by index is fast. However, updating or adding elements to tuples is expensive because it requires copying the whole tuple in memory.

When “counting” the number of elements in a data structure, Elixir also abides by a simple rule: the function is named size if the operation is in constant time (i.e. the value is pre-calculated) or length if the operation is linear (i.e. calculating the length gets slower as the input grows).

Pattern Matching


The match operator is not only used to match against simple values, but it is also useful for destructuring more complex data types. For example, we can pattern match on tuples:

iex> {a, b, c} = {:hello, "world", 42}
{:hello, "world", 42}
iex> a
:hello
iex> b
"world"

A list also supports matching on its own head and tail:

iex> [head | tail] = [1, 2, 3]
[1, 2, 3]
iex> head
1
iex> tail
[2, 3]

The pin operator ^ should be used when you want to pattern match against an existing variable’s value rather than rebinding the variable.

iex> x = 1
1
iex> ^x = 2
** (MatchError) no match of right hand side value: 2
iex> {y, ^x} = {2, 1}
{2, 1}
iex> y
2
iex> {y, ^x} = {2, 2}
** (MatchError) no match of right hand side value: {2, 2}

case, cond and if


case

Behavious pretty much as the classic case statement.

iex> case {1, 2, 3} do
...>   {4, 5, 6} ->
...>     "This clause won't match"
...>   {1, x, 3} ->
...>     "This clause will match and bind x to 2 in this clause"
...>   _ ->
...>     "This clause would match any value"
...> end

If you want to pattern match against an existing variable, you need to use the ^ operator:

iex> x = 1
1
iex> case 10 do
...>   ^x -> "Won't match"
...>   _  -> "Will match"
...> end

Another cool example, now with clauses conditions:

iex> case {1, 2, 3} do
...>   {1, x, 3} when x > 0 ->
...>     "Will match"
...>   _ ->
...>     "Would match, if guard condition were not satisfied"
...> end
  • If none of the clauses match, an error is raised.

cond

case is useful when you need to match against different values. However, in many circumstances, we want to check different conditions and find the first one that evaluates to true. In such cases, one may use cond.

iex> cond do
...>   2 + 2 == 5 ->
...>     "This will not be true"
...>   2 * 2 == 3 ->
...>     "Nor this"
...>   1 + 1 == 2 ->
...>     "But this will"
...> end
  • This is equivalent to else and if clauses in many imperative languages.
  • If none of the conditions return true, an error is raised. For this reason, it may be necessary to add a final condition, equal to true, which will always match.

if and unless

Are useful when you need to check for just one condition, also pro provides a else statement.

iex> if true do
...>   "This works!"
...> end
"This works!"
iex> unless true do
...>   "This will never be seen"
...> end
nil

do / end blocks

Equivalent to { / }, it’s also possible things like:

iex> if false, do: :this, else: :that
# Expressions like:
iex> is_number if true do
...>  1 + 2
...> end
** (CompileError) undefined function: is_number/2
# Should be:
iex> is_number(if true do
...>  1 + 2
...> end)
true

Binaries, strings and char lists


Binaries and bitstrings

You can define a binary using <<>>. It’s just a sequence of bytes. The string concatenation operation is actually a binary concatenation operator <>.

A common trick in Elixir is to concatenate the null byte <<0>> to a string to see its inner binary representation:

iex> "hełło" <> <<0>>
<<104, 101, 197, 130, 197, 130, 111, 0>>

A binary is a bitstring where the number of bits is divisible by 8. Smaller bit are just bitstrings!

A string is a UTF-8 encoded binary, and a binary is a bitstring where the number of bits is divisible by 8.

Char lists

A char list is nothing more than a list of characters.

Char list contains the code points of the characters between single-quotes (note that IEx will only output code points if any of the chars is outside the ASCII range). So while double-quotes represent a string (i.e. a binary), single-quotes represents a char list (i.e. a list).

Keywords and maps


Keyword list

It’s a associative data structure. In Elixir, when we have a list of tuples and the first item of the tuple (i.e. the key) is an atom, we call it a keyword list:

iex> list = [{:a, 1}, {:b, 2}]
[a: 1, b: 2]
iex> list == [a: 1, b: 2]
true
iex> list[:a]
1
  • It’s the default mechanism for passing options to functions in Elixir.
  • Only allows Atoms as keys.
  • Ordered as specified by the developer.
  • Remember, though, keyword lists are simply lists, and as such they provide the same linear performance characteristics as lists. The longer the list, the longer it takes to read from. For bigger data use maps instead.

Maps

Whenever you need a key-value store, maps are the “go to” data structure in Elixir:

iex> map = %{:a => 1, 2 => :b}
%{2 => :b, :a => 1}
iex> map[:a]
1
iex> map[2]
:b
iex> map[:c]
nil
  • Allows any value as key.
  • Maps’ keys do not follow any ordering.

Interacts great with pattern matching:

iex> %{} = %{:a => 1, 2 => :b}
%{:a => 1, 2 => :b}
iex> %{:a => a} = %{:a => 1, 2 => :b}
%{:a => 1, 2 => :b}
iex> a
1
iex> %{:c => c} = %{:a => 1, 2 => :b}
** (MatchError) no match of right hand side value: %{2 => :b, :a => 1}

Better syntax when all keys are atoms:

iex> map = %{a: 1, b: 2}

Another interesting property of maps is that they provide their own syntax for updating and accessing atom keys:

iex> map = %{:a => 1, 2 => :b}
%{:a => 1, 2 => :b}
iex> map.a
1
iex> map.c
** (KeyError) key :c not found in: %{2 => :b, :a => 1}
iex> %{map | :a => 2}
%{:a => 2, 2 => :b}
iex> %{map | :c => 3}
** (KeyError) key :c not found in: %{2 => :b, :a => 1}

Nested data structures (put_in/2 and update_in/2)

iex> users = [
  john: %{name: "John", age: 27, languages: ["Erlang", "Ruby", "Elixir"]},
  mary: %{name: "Mary", age: 29, languages: ["Elixir", "F#", "Clojure"]}
]

We have a keyword list of users where each value is a map containing the name, age and a list of programming languages each user likes. If we wanted to access the age for john, we could write:

iex> users[:john].age
27

It happens we can also use this same syntax for updating the value:

iex> users = put_in users[:john].age, 31
[john: %{age: 31, languages: ["Erlang", "Ruby", "Elixir"], name: "John"},
 mary: %{age: 29, languages: ["Elixir", "F#", "Clojure"], name: "Mary"}]

The update_in/2 macro is similar but allows us to pass a function that controls how the value changes. For example, let’s remove “Clojure” from Mary’s list of languages:

iex> users = update_in users[:mary].languages, &List.delete(&1, "Clojure")
[john: %{age: 31, languages: ["Erlang", "Ruby", "Elixir"], name: "John"},
mary: %{age: 29, languages: ["Elixir", "F#"], name: "Mary"}]

There is more to learn about put_in/2 and update_in/2, including the get_and_update_in/2 that allows us to extract a value and update the data structure at once. There are also put_in/3, update_in/3 and get_and_update_in/3 which allow dynamic access into the data structure. Check their respective documentation in the Kernel module for more information.

Modules


In Elixir we group several functions into modules.

iex> defmodule Math do
...>   def sum(a, b) do
...>     a + b
...>   end
...> end

iex> Math.sum(1, 2)
3

Compilation

Given a file math.ex:

defmodule Math do
  def sum(a, b) do
    a + b
  end
end

This file can be compiled using elixirc:

$ elixirc math.ex

This will generate a file named Elixir.Math.beam containing the bytecode for the defined module. If we start iex again, our module definition will be available (provided that iex is started in the same directory the bytecode file is in).

Elixir projects are usually organized into three directories:

  • ebin - contains the compiled bytecode
  • lib - contains elixir code (usually .ex files)
  • test - contains tests (usually .exs files)

When working on actual projects, the build tool called mix will be responsible for compiling and setting up the proper paths for you.

Scripted mode

  • .ex - files to be compiled
  • .exs - files to run in scripted mode (Learning purposes)

Executing:

$ elixir math.exs

Named functions

  • def/2 - defines a function
  • defp/2 - defines a private function

Function declarations also support guards and multiple clauses. If a function has several clauses, Elixir will try each clause until it finds one that matches.

defmodule Math do
  def zero?(0) do
    true
  end

  def zero?(x) when is_integer(x) do
    false
  end
end

IO.puts Math.zero?(0)         #=> true
IO.puts Math.zero?(1)         #=> false
IO.puts Math.zero?([1, 2, 3]) #=> ** (FunctionClauseError)
IO.puts Math.zero?(0.0)       #=> ** (FunctionClauseError)

Similar to constructs like if, named functions support both do: and do/end block syntax, as we learned do/end is just a convenient syntax for the keyword list format. For example, we can edit math.exs to look like this:

defmodule Math do
  def zero?(0), do: true
  def zero?(x) when is_integer(x), do: false
end

Function capturing

Can actually be used to retrieve a named function as a function type. (Given the file)

iex> Math.zero?(0)
true
iex> fun = &Math.zero?/1
&Math.zero?/1
iex> is_function(fun)
true
iex> fun.(0)
true

Local or imported functions, like is_function/1, can be captured without the module:

iex> &is_function/1
&:erlang.is_function/1
iex> (&is_function/1).(fun)
true

Note the capture syntax can also be used as a shortcut for creating functions:

iex> fun = &(&1 + 1)
#Function<6.71889879/1 in :erl_eval.expr/5>
iex> fun.(1)
2

The &1 represents the first argument passed into the function.
The above is the same as fn x -> x + 1 end. It’s useful for short function definitions.

If you want to capture a function from a module, you can do &Module.function():

iex> fun = &List.flatten(&1, &2)
&List.flatten/2
iex> fun.([1, [[2], 3]], [4, 5])
[1, 2, 3, 4, 5]

&List.flatten(&1, &2) is the same as writing fn(list, tail) -> List.flatten(list, tail) end which in this case is equivalent to &List.flatten/2. You can read more about the capture operator & in the Kernel.SpecialForms documentation.

Default arguments

Named functions default arguments:

defmodule Concat do
  def join(a, b, sep \\ " ") do
    a <> sep <> b
  end
end

IO.puts Concat.join("Hello", "world")      #=> Hello world
IO.puts Concat.join("Hello", "world", "_") #=> Hello_world

If a function with default values has multiple clauses, it is required to create a function head (without an actual body) for declaring defaults:

defmodule Concat do
  def join(a, b \\ nil, sep \\ " ")

  def join(a, b, _sep) when is_nil(b) do
    a
  end

  def join(a, b, sep) do
    a <> sep <> b
  end
end

IO.puts Concat.join("Hello", "world")      #=> Hello world
IO.puts Concat.join("Hello", "world", "_") #=> Hello_world
IO.puts Concat.join("Hello")               #=> Hello
  • Default values won’t be evaluated during the function definition
  • When using default values, one must be careful to avoid overlapping function definitions

Recursion


Loops through recursion

Beautifully without mutating:

defmodule Recursion do
  def print_multiple_times(msg, n) when n <= 1 do
    IO.puts msg
  end

  def print_multiple_times(msg, n) do
    IO.puts msg
    print_multiple_times(msg, n - 1)
  end
end

Recursion.print_multiple_times("Hello!", 3)
# Hello!
# Hello!
# Hello!

Reduce and map algorithms

Let’s now see how we can use the power of recursion to sum a list of numbers:

defmodule Math do
  def sum_list([head | tail], accumulator) do
    sum_list(tail, head + accumulator)
  end

  def sum_list([], accumulator) do
    accumulator
  end
end

IO.puts Math.sum_list([1, 2, 3], 0) #=> 6

The process of taking a list and reducing it down to one value is known as a reduce algorithm and is central to functional programming.

What if we instead want to double all of the values in our list?

defmodule Math do
  def double_each([head | tail]) do
    [head * 2 | double_each(tail)]
  end

  def double_each([]) do
    []
  end
end
$ iex math.exs
iex> Math.double_each([1, 2, 3]) #=> [2, 4, 6]

Here we have used recursion to traverse a list, doubling each element and returning a new list. The process of taking a list and mapping over it is known as a map algorithm.

Recursion and tail call optimization are an important part of Elixir. However, when programming in Elixir you will rarely use recursion as above. The Enum module, (next chapter), already provides many conveniences for working with lists. For instance, the examples above could be written as:

iex> Enum.reduce([1, 2, 3], 0, fn(x, acc) -> x + acc end)
6
iex> Enum.map([1, 2, 3], fn(x) -> x * 2 end)
[2, 4, 6]

Or, using the capture syntax:

iex> Enum.reduce([1, 2, 3], 0, &+/2)
6
iex> Enum.map([1, 2, 3], &(&1 * 2))
[2, 4, 6]

Enumerables and Streams


The Enum module provides a huge range of functions to transform, sort, group, filter and retrieve items from enumerables. It is one of the modules developers use frequently in their Elixir code.

For specific operations, like inserting and updating particular elements, you may need to reach for modules specific to the data type. For example, if you want to insert an element at a given position in a list, you should use the List.insert_at/3 function from the List module.

We say the functions in the Enum module are polymorphic because they can work with diverse data types. In particular, the functions in the Enum module can work with any data type that implements the Enumerable protocol.

BTW, Elixir also provides ranges:

iex> Enum.map(1..3, fn x -> x * 2 end)
[2, 4, 6]
iex> Enum.reduce(1..3, 0, &+/2)
6

Eager vs Lazy

All the functions in the Enum module are eager. Many functions expect an enumerable and return a list back. This means that when performing multiple operations with Enum, each operation is going to generate an intermediate list until we reach the result:

iex> 1..100_000 |> Enum.map(&(&1 * 3)) |> Enum.filter(odd?) |> Enum.sum
7500000000

The pipe operator

The |> symbol used in the snippet above is the pipe operator: it simply takes the output from the expression on its left side and passes it as the first argument to the function call on its right side. It’s similar to the Unix | operator.

Streams

As an alternative to Enum, Elixir provides the Stream module which supports lazy operations:

iex> 1..100_000 |> Stream.map(&(&1 * 3)) |> Stream.filter(odd?) |> Enum.sum
7500000000

In the example above, 1..100_000 |> Stream.map(&(&1 * 3)) returns a data type, an actual stream, that represents the map computation over the range 1..100_000. Furthermore, they are composable because we can pipe many stream operations:

iex> 1..100_000 |> Stream.map(&(&1 * 3)) |> Stream.filter(odd?)
#Stream<[enum: 1..100000, funs: [...]]>

Instead of generating intermediate lists, streams build a series of computations that are invoked only when we pass the underlying stream to the Enum module. Streams are useful when working with large, possibly infinite, collections.

It also provides functions for creating streams. For example, Stream.cycle/1 can be used to create a stream that cycles a given enumerable infinitely:

iex> stream = Stream.cycle([1, 2, 3])
#Function<15.16982430/2 in Stream.cycle/1>
iex> Enum.take(stream, 10)
[1, 2, 3, 1, 2, 3, 1, 2, 3, 1]

Another interesting function is Stream.resource/3 which can be used to wrap around resources, guaranteeing they are opened right before enumeration and closed afterwards, even in the case of failures. For example, we can use it to stream a file:

iex> stream = File.stream!("path/to/file")
#Function<18.16982430/2 in Stream.resource/3>
iex> Enum.take(stream, 10)

The example above will fetch the first 10 lines of the file you have selected. This means streams can be very useful for handling large files or even slow resources like network resources.

Processes


In Elixir, all code runs inside processes. Processes are isolated from each other, run concurrent to one another and communicate via message passing.

spawn

The basic mechanism for spawning new processes is with the auto-imported spawn/1 function:

iex> pid = spawn fn -> 1 + 2 end
#PID<0.43.0>
iex> Process.alive?(pid)
false

We can retrieve the PID of the current process by calling self/0.

send and receive

We can send messages to a process with send/2 and receive them with receive/1:

iex> send self(), {:hello, "world"}
{:hello, "world"}
iex> receive do
...>   {:hello, msg} -> msg
...>   {:world, msg} -> "won't match"
...> end
"world"

When a message is sent to a process, the message is stored in the process mailbox. The receive/1 block goes through the current process mailbox searching for a message that matches any of the given patterns. receive/1 supports guards and many clauses, such as case/2.

If there is no message in the mailbox matching any of the patterns, the current process will wait until a matching message arrives. A timeout can also be specified (A timeout of 0 can be given when you already expect the message to be in the mailbox):

iex> receive do
...>   {:hello, msg}  -> msg
...> after
...>   1_000 -> "nothing after 1s"
...> end
"nothing after 1s"

Let’s put it all together and send messages between processes:

iex> parent = self()
#PID<0.41.0>
iex> spawn fn -> send(parent, {:hello, self()}) end
#PID<0.48.0>
iex> receive do
...>   {:hello, pid} -> "Got hello from #{inspect pid}"
...> end
"Got hello from #PID<0.48.0>

While in the shell, you may find the helper flush/0 quite useful. It flushes and prints all the messages in the mailbox.

iex> send self(), :hello
:hello
iex> flush()
:hello
:ok

The most common form of spawning in Elixir is actually via spawn_link/1. Before we show an example with spawn_link/1, let’s try to see what happens when a process fails:

iex> spawn fn -> raise "oops" end
#PID<0.58.0>

[error] Process #PID<0.58.00> raised an exception
** (RuntimeError) oops
    :erlang.apply/2

It merely logged an error but the spawning process is still running. That’s because processes are isolated. If we want the failure in one process to propagate to another one, we should link them. This can be done with spawn_link/1:

iex> spawn_link fn -> raise "oops" end
#PID<0.41.0>

** (EXIT from #PID<0.41.0>) an exception was raised:
    ** (RuntimeError) oops
        :erlang.apply/2

In Elixir applications, we often link our processes to supervisors which will detect when a process dies and start a new process in its place. This is only possible because processes are isolated and don’t share anything by default. And since processes are isolated, there is no way a failure in a process will crash or corrupt the state of another.

While other languages would require us to catch/handle exceptions, in Elixir we are actually fine with letting processes fail because we expect supervisors to properly restart our systems. “Failing fast” is a common philosophy when writing Elixir software!

spawn/1 and spawn_link/1 are the basic primitives for creating processes in Elixir. Although we have used them exclusively so far, most of the time we are going to use abstractions that build on top of them. Let’s see the most common one, called tasks.

Tasks

Tasks build on top of the spawn functions to provide better error reports and introspection:

iex(1)> Task.start fn -> raise "oops" end
{:ok, #PID<0.55.0>}

15:22:33.046 [error] Task #PID<0.55.0> started from #PID<0.53.0> terminating
** (RuntimeError) oops
    (elixir) lib/task/supervised.ex:74: Task.Supervised.do_apply/2
    (stdlib) proc_lib.erl:239: :proc_lib.init_p_do_apply/3
Function: #Function<20.90072148/0 in :erl_eval.expr/5>
    Args: []

Instead of spawn/1 and spawn_link/1, we use Task.start/1 and Task.start_link/1 to return {:ok, pid} rather than just the PID. This is what enables Tasks to be used in supervision trees. Furthermore, Task provides convenience functions, like Task.async/1 and Task.await/1, and functionality to ease distribution.

State

We haven’t talked about state so far in this guide. If you are building an application that requires state, for example, to keep your application configuration, or you need to parse a file and keep it in memory, where would you store it?

Processes are the most common answer to this question. We can write processes that loop infinitely, maintain state, and send and receive messages. As an example, let’s write a module that starts new processes that work as a key-value store in a file named kv.exs:

defmodule KV do
  def start_link do
    Task.start_link(fn -> loop(%{}) end)
  end

  defp loop(map) do
    receive do
      {:get, key, caller} ->
        send caller, Map.get(map, key)
        loop(map)
      {:put, key, value} ->
        loop(Map.put(map, key, value))
    end
  end
end

Let’s give it a try by running $ iex kv.exs:

iex> {:ok, pid} = KV.start_link
#PID<0.62.0>
iex> send pid, {:get, :hello, self()}
{:get, :hello, #PID<0.41.0>}
iex> flush
nil
:ok

At first, the process map has no keys, so sending a :get message and then flushing the current process inbox returns nil. Let’s send a :put message and try it again:

iex> send pid, {:put, :hello, :world}
{:put, :hello, :world}
iex> send pid, {:get, :hello, self()}
{:get, :hello, #PID<0.41.0>}
iex> flush
:world
:ok

Notice how the process is keeping a state and we can get and update this state by sending the process messages. In fact, any process that knows the pid above will be able to send it messages and manipulate the state.

It is also possible to register the pid, giving it a name, and allowing everyone that knows the name to send it messages:

iex> Process.register(pid, :kv)
true
iex> send :kv, {:get, :hello, self()}
{:get, :hello, #PID<0.41.0>}
iex> flush
:world
:ok

Using processes around state and name registering are very common patterns in Elixir applications. However, most of the time, we won’t implement those patterns manually as above, but by using one of the many abstractions that ship with Elixir. For example, Elixir provides agents, which are simple abstractions around state:

iex> {:ok, pid} = Agent.start_link(fn -> %{} end)
{:ok, #PID<0.72.0>}
iex> Agent.update(pid, fn map -> Map.put(map, :hello, :world) end)
:ok
iex> Agent.get(pid, fn map -> Map.get(map, :hello) end)
:world

A :name option could also be given to Agent.start_link/2 and it would be automatically registered. Besides agents, Elixir provides an API for building generic servers (called GenServer), tasks and more, all powered by processes underneath. Those, along with supervision trees, will be explored with more detail in the Mix and OTP guide which will build a complete Elixir application from start to finish.

IO and the file system


The IO module is the main mechanism in Elixir for reading and writing to standard input/output (:stdio), standard error (:stderr), files and other IO devices. Usage of the module is pretty straightforward:

iex> IO.puts "hello world"
hello world
:ok
iex> IO.gets "yes or no? "
yes or no? yes
"yes\n"

By default, functions in the IO module read from the standard input and write to the standard output. We can change that by passing, for example, :stderr as an argument (in order to write to the standard error device):

iex> IO.puts :stderr, "hello world"
hello world
:ok

The File module

The File module contains functions that allow us to open files as IO devices. By default, files are opened in binary mode, which requires developers to use the specific IO.binread/2 and IO.binwrite/2 functions from the IO module:

iex> {:ok, file} = File.open "hello", [:write]
{:ok, #PID<0.47.0>}
iex> IO.binwrite file, "world"
:ok
iex> File.close file
:ok
iex> File.read "hello"
{:ok, "world"}

A file can also be opened with :utf8 encoding, which tells the File module to interpret the bytes read from the file as UTF-8-encoded bytes.

It also provides Unix like functions: File.rm/1, File.mkdir/1, File.mkdir_p/1, etc. (Checkout the module documentation) Notice the variations with a trailing bang !.

iex> File.read "hello"
{:ok, "world"}
iex> File.read! "hello"
"world"
iex> File.read "unknown"
{:error, :enoent}
iex> File.read! "unknown"
** (File.Error) could not read file unknown: no such file or directory

Notice that when the file does not exist, the version with ! raises an error. The version without ! is preferred when you want to handle different outcomes using pattern matching:

case File.read(file) do
  {:ok, body}      -> # do something with the `body`
  {:error, reason} -> # handle the error caused by `reason`
end

However, if you expect the file to be there, the bang variation is more useful as it raises a meaningful error message. Avoid writing: {:ok, body} = File.read(file) as, in case of an error, File.read/1 will return {:error, reason} and the pattern matching will fail. You will still get the desired result (a raised error), but the message will be about the pattern which doesn’t match (thus being cryptic in respect to what the error actually is about).

Therefore, if you don’t want to handle the error outcomes, prefer using File.read!/1.

The Path module

The majority of the functions in the File module expect paths as arguments. Most commonly, those paths will be regular binaries. The Path module provides facilities for working with such paths:

iex> Path.join("foo", "bar")
"foo/bar"
iex> Path.expand("~/hello")
"/Users/jose/hello"

Processes and group leaders

By modelling IO devices with processes, the Erlang VM allows different nodes in the same network to exchange file processes in order to read/write files in between nodes. Of all IO devices, there is one that is special to each process: the group leader.

When you write to :stdio, you are actually sending a message to the group leader, which writes to the standard-output file descriptor:

iex> IO.puts :stdio, "hello"
hello
:ok
iex> IO.puts Process.group_leader, "hello"
hello
:ok

The group leader can be configured per process and is used in different situations. For example, when executing code in a remote terminal, it guarantees messages in a remote node are redirected and printed in the terminal that triggered the request.

alias, require and import


# Alias the module so it can be called as Bar instead of Foo.Bar
alias Foo.Bar, as: Bar

# Ensure the module is compiled and available (usually for macros)
require Foo

# Import functions from Foo so they can be called without the `Foo.` prefix
import Foo

# Invokes the custom code defined in Foo as an extension point
use Foo

We are going to explore them in detail now. Keep in mind the first three are called directives because they have lexical scope , while use is a common extension point.

alias

defmodule Math do
  alias Math.List, as: List
end

From now on, any reference to List will automatically expand to Math.List. In case one wants to access the original List, it can be done by prefixing the module name with Elixir.:

List.flatten             #=> uses Math.List.flatten
Elixir.List.flatten      #=> uses List.flatten
Elixir.Math.List.flatten #=> uses Math.List.flatten

Note: All modules defined in Elixir are defined inside a main Elixir namespace. However, for convenience, you can omit “Elixir.” when referencing them.

Note that alias is lexically scoped, which allows you to set aliases inside specific functions:

defmodule Math do
  def plus(a, b) do
    alias Math.List
    # ...
  end

  def minus(a, b) do
    # ...
  end
end

require

In order to use a macro, we need to guarantee its module and implementation are available during compilation. This is done with the require directive:

iex> Integer.is_odd(3)
** (CompileError) iex:1: you must require Integer before invoking the macro Integer.is_odd/1
iex> require Integer
Integer
iex> Integer.is_odd(3)
true

Note that like the alias directive, require is also lexically scoped.

import

We use import whenever we want to easily access functions or macros from other modules without using the fully-qualified name. For instance, if we want to use the duplicate/2 function from the List module several times, we can import it:

iex> import List, only: [duplicate: 2]
List
iex> duplicate :ok, 3
[:ok, :ok, :ok]

:except could also be given as an option. import also supports :macros and :functions to be given to :only. For example, to import all macros, one could write: import Integer, only: :macros.

Note that import is lexically scoped too. This means that we can import specific macros or functions inside function definitions:

defmodule Math do
  def some_function do
    import List, only: [duplicate: 2]
    duplicate(:ok, 10)
  end
end

Note that importing a module automatically requires it.

use

Although not a directive, use is a macro tightly related to require that allows you to use a module in the current context. The use macro is frequently used by developers to bring external functionality into the current lexical scope, often modules.

For example, in order to write tests using the ExUnit framework, a developer should use the ExUnit.Case module:

defmodule AssertionTest do
  use ExUnit.Case, async: true

  test "always pass" do
    assert true
  end
end

Module attributes


  • They serve to annotate the module, often with information to be used by the user or the VM.
  • They work as constants.
  • They work as a temporary module storage to be used during compilation.

As annotations

Elixir has a handful of reserved attributes. Here are a few of them, the most commonly used ones:

  • @moduledoc - provides documentation for the current module.
  • @doc - provides documentation for the function or macro that follows the attribute.
  • @behaviour - (notice the British spelling) used for specifying an OTP or user-defined behaviour.
  • @before_compile - provides a hook that will be invoked before the module is compiled. This makes it possible to inject functions inside the module exactly before compilation.

As constants

defmodule MyServer do
  @initial_state %{host: "147.0.0.1", port: 3456}
  IO.inspect @initial_state
end

Note: Unlike Erlang, user defined attributes are not stored in the module by default. The value exists only during compilation time. A developer can configure an attribute to behave closer to Erlang by calling Module.register_attribute/3.

Attributes can also be read inside functions:

defmodule MyServer do
  @my_data 14
  def first_data, do: @my_data
  @my_data 13
  def second_data, do: @my_data
end

MyServer.first_data #=> 14
MyServer.second_data #=> 13

Every time an attribute is read inside a function, a snapshot of its current value is taken. In other words, the value is read at compilation time and not at runtime.

As temporary storage

Attributes can be used to do so. The ExUnit framework which uses module attributes as annotation and storage:

defmodule MyTest do
  use ExUnit.Case

  @tag :external
  test "contacts external service" do
    # ...
  end
end

Structs


Structs are extensions built on top of maps that provide compile-time checks and default values.

iex> defmodule User do
...>   defstruct name: "John", age: 27
...> end

Structs take the name of the module they’re defined in. In the example above, we defined a struct named User. Let’s create one so:

iex> %User{}
%User{age: 27, name: "John"}
iex> %User{name: "Meg"}
%User{age: 27, name: "Meg"}

To access and update:

iex> john = %User{}
%User{age: 27, name: "John"}
iex> john.name
"John"
iex> meg = %{john | name: "Meg"}
%User{age: 27, name: "Meg"}
iex> %{meg | oops: :field}
** (KeyError) key :oops not found in: %User{age: 27, name: "Meg"}

When using the update syntax (|), the VM is aware that no new keys will be added to the struct, allowing the maps underneath to share their structure in memory.

Structs can also be used in pattern matching, both for matching on the value of specific keys as well as for ensuring that the matching value is a struct of the same type as the matched value.

iex> %User{name: name} = john
%User{age: 27, name: "John"}
iex> name
"John"
iex> %User{} = %{}
** (MatchError) no match of right hand side value: %{}

Note: Structs are bare maps underneath. As maps, structs store a “special” field named __struct__ that holds the name of the struct. We referred to structs as bare maps because none of the protocols implemented for maps are available for structs. For example, you can neither enumerate nor access a struct. However, since structs are just maps, they work with the functions from the Map module.

Structs alongside protocols provide one of the most important features for Elixir developers: data polymorphism.

Protocols


Protocols are a mechanism to achieve polymorphism in Elixir. Dispatching on a protocol is available to any data type as long as it implements the protocol.

Let’s implement that to specify a blank? protocol that returns a boolean for other data types that should be considered blank.

defprotocol Blank do
  @doc "Returns true if data is considered blank/empty"
  def blank?(data)
end

The protocol expects a function called blank? that receives one argument to be implemented. We can implement this protocol for different Elixir data types as follows:

# Integers are never blank
defimpl Blank, for: Integer do
  def blank?(_), do: false
end

# Just empty list is blank
defimpl Blank, for: List do
  def blank?([]), do: true
  def blank?(_),  do: false
end

# Just empty map is blank
defimpl Blank, for: Map do
  # Keep in mind we could not pattern match on %{} because
  # it matches on all maps. We can however check if the size
  # is zero (and size is a fast operation).
  def blank?(map), do: map_size(map) == 0
end

# Just the atoms false and nil are blank
defimpl Blank, for: Atom do
  def blank?(false), do: true
  def blank?(nil),   do: true
  def blank?(_),     do: false
end
iex> Blank.blank?(0)
false
iex> Blank.blank?([])
true
iex> Blank.blank?([1, 2, 3])
false

And we would do so for all native data types. The types available are:

  • Atom
  • BitString
  • Float
  • Function
  • Integer
  • List
  • Map
  • PID
  • Port
  • Reference
  • Tuple

Manually implementing protocols for all types can quickly become repetitive and tedious. In such cases, Elixir provides two options: we can explicitly derive the protocol implementation for our types or automatically implement the protocol for all types. In both cases, we need to implement the protocol for Any.

Deriving

defimpl Blank, for: Any do
  def blank?(_), do: false
end
defmodule DeriveUser do
  @derive Blank
  defstruct name: "john", age: 27
end

Fallback to Any

Another alternative to @derive is to explicitly tell the protocol to fallback to Any when an implementation cannot be found. This can be achieved by setting @fallback_to_any to true in the protocol definition:

defprotocol Blank do
  @fallback_to_any true
  def blank?(data)
end

Comprehensions


Declared by for:

# Map a list of integers into their squared values.
iex> for n <- [1, 2, 3, 4], do: n * n
[1, 4, 9, 16]

A comprehension is made of three parts: generators, filters and collectables.

Generators and filters

In the expression above, n <- [1, 2, 3, 4] is the generator. It is literally generating values to be used in the comprehension. Any enumerable can be passed in the right-hand side of the generator expression:

iex> for n <- 1..4, do: n * n
[1, 4, 9, 16]

It also supports pattern matching:

iex> values = [good: 1, good: 2, bad: 3, good: 4]
iex> for {:good, n} <- values, do: n * n
[1, 4, 16]

Alternatively to pattern matching, filters can be used to select some particular elements:

iex> multiple_of_3? = fn(n) -> rem(n, 3) == 0 end
iex> for n <- 0..5, multiple_of_3?.(n), do: n * n
[0, 9]

Comprehensions also allow multiple generators and filters to be given. Here is an example that receives a list of directories and gets the size of each file in those directories:

for dir  <- dirs,
    file <- File.ls!(dir),
    path = Path.join(dir, file),
    File.regular?(path) do
  File.stat!(path).size
end

Calculating the cartesian product of two lists:

iex> for i <- [:a, :b, :c], j <- [1, 2], do:  {i, j}
[a: 1, a: 2, b: 1, b: 2, c: 1, c: 2]

A more advanced example of multiple generators and filters is Pythagorean triples. A Pythagorean triple is a set of positive integers such that a*a + b*b = c*c, let’s write a comprehension in a file named triple.exs:

defmodule Triple do
  def pythagorean(n) when n > 0 do
    for a <- 1..n,
        b <- 1..n,
        c <- 1..n,
        a + b + c <= n,
        a*a + b*b == c*c,
        do: {a, b, c}
  end
end

Outputs:

$ iex triple.exs
iex> Triple.pythagorean(5)
[]
iex> Triple.pythagorean(12)
[{3, 4, 5}, {4, 3, 5}]
iex> Triple.pythagorean(48)
[{3, 4, 5}, {4, 3, 5}, {5, 12, 13}, {6, 8, 10}, {8, 6, 10}, {8, 15, 17},
 {9, 12, 15}, {12, 5, 13}, {12, 9, 15}, {12, 16, 20}, {15, 8, 17}, {16, 12, 20}]

Take a closer look on how it performs without filters:

 defmodule Triple do
  def pythagorean(n) when n > 0 do
    for a <- 1..n,
        b <- 1..n,
        c <- 1..n,
        do: IO.puts "#{a} #{b} #{c}"
  end
end

Triple.pythagorean(10)

Outputs:

$ elixir triple.exs
1 1 1
1 1 2
1 1 3
1 1 4
1 1 5
1 1 6
1 1 7
1 1 8
1 1 9
1 1 10
1 2 1
1 2 2
[...]

Sigils


Is one of the mechanisms provided by the language for working with textual representations (also allowing extensibility). Sigils start with the tilde (~) character which is followed by a letter (which identifies the sigil) and then a delimiter; optionally, modifiers can be added after the final delimiter.

The most common sigil in Elixir is ~r, which is used to create regular expressions:

# A regular expression that matches strings which contain "foo" or "bar":
iex> regex = ~r/foo|bar/
~r/foo|bar/
iex> "foo" =~ regex
true
iex> "bat" =~ regex
false

So far, all examples have used / to delimit a regular expression. However sigils support 8 different delimiters:

~r/hello/
~r|hello|
~r"hello"
~r'hello'
~r(hello)
~r[hello]
~r{hello}
~r<hello>

Strings

The ~s sigil is used to generate strings, like double quotes are. The ~s sigil is useful, for example, when a string contains both double and single quotes:

iex> ~s(this is a string with "double" quotes, not 'single' ones)
"this is a string with \"double\" quotes, not 'single' ones"

Char lists

iex> ~c(this is a char list containing 'single quotes')
'this is a char list containing \'single quotes\''

Word lists (words are just regular strings)

iex> ~w(foo bar bat)
["foo", "bar", "bat"]

The ~w sigil also accepts the c, s and a modifiers (for char lists, strings and atoms, respectively), which specify the data type of the elements of the resulting list:

iex> ~w(foo bar bat)a
[:foo, :bar, :bat]

Interpolation and escaping in sigils

Besides lowercase sigils, Elixir supports uppercase sigils to deal with escaping characters and interpolation.

iex> ~s(String with escape codes \x26 #{"inter" <> "polation"})
"String with escape codes & interpolation"
iex> ~S(String without escape codes \x26 without #{interpolation})
"String without escape codes \\x26 without \#{interpolation}"

The following escape codes can be used in strings and char lists:

  • \" – double quote
  • \' – single quote
  • \\ – single backslash
  • \a– bell/alert
  • \b – backspace
  • \d - delete
  • \e - escape
  • \f - form feed
  • \n – newline
  • \r – carriage return
  • \s – space
  • \t – tab
  • \v – vertical tab
  • \0 - null byte
  • \xDD - represents a single byte in hexadecimal (such as \x13)
  • \uDDDD and \u{D...} - represents a Unicode codepoint in hexadecimal (such as \u{1F600})

Also supports herecods:

iex> ~s"""
...> this is
...> a heredoc string
...> """

Writing escape characters in documentation would soon become error prone because of the need to double-escape some characters. By using ~S, this problem can be avoided altogether:

@doc ~S"""
Converts double-quotes to single-quotes.

## Examples

    iex> convert("\"foo\"")
    "'foo'"

"""
def convert(...)

Custom sigils

We can also provide our own sigils by implementing functions that follow the sigil_{identifier} pattern. For example, let’s implement the ~i sigil that returns an integer (with the optional n modifier to make it negative):

iex> defmodule MySigils do
...>   def sigil_i(string, []), do: String.to_integer(string)
...>   def sigil_i(string, [?n]), do: -String.to_integer(string)
...> end
iex> import MySigils
iex> ~i(13)
13
iex> ~i(42)n
-42

Sigils can also be used to do compile-time work with the help of macros. For example, regular expressions in Elixir are compiled into an efficient representation during compilation of the source code, therefore skipping this step at runtime. If you’re interested in the subject, we recommend you learn more about macros and check out how sigils are implemented in the Kernel module (where the sigil_* functions are defined).

try, catch and rescue


Errors

Errors (or exceptions) are used when exceptional things happen in the code. A sample error can be retrieved by trying to add a number into an atom:

iex> :foo + 1
** (ArithmeticError) bad argument in arithmetic expression
     :erlang.+(:foo, 1)

A runtime error can be raised any time by using raise/1:

iex> raise "oops"
** (RuntimeError) oops

Other errors can be raised with raise/2 passing the error name and a list of keyword arguments:

iex> raise ArgumentError, message: "invalid argument foo"
** (ArgumentError) invalid argument fo

You can also define your own errors by creating a module and using the defexception construct inside it; this way, you’ll create an error with the same name as the module it’s defined in. The most common case is to define a custom exception with a message field:

iex> defmodule MyError do
iex>   defexception message: "default message"
iex> end
iex> raise MyError
** (MyError) default message
iex> raise MyError, message: "custom message"
** (MyError) custom message

Errors can be rescued using the try/rescue construct:

iex> try do
...>   raise "oops"
...> rescue
...>   e in RuntimeError -> e
...> end
%RuntimeError{message: "oops"}

If you don’t have any use for the error, you don’t have to provide it:

iex> try do
...>   raise "oops"
...> rescue
...>   RuntimeError -> "Error!"
...> end
"Error!"

In Elixir, we avoid using try/rescue because we don’t use errors for control flow. We take errors literally: they are reserved for unexpected and/or exceptional situations. In case you actually need flow control constructs, throws should be used. That’s what we are going to see next.

Throws

In Elixir, a value can be thrown and later be caught. throw and catch are reserved for situations where it is not possible to retrieve a value unless by using throw and catch.

Those situations are quite uncommon in practice except when interfacing with libraries that do not provide a proper API. For example, let’s imagine the Enum module did not provide any API for finding a value and that we needed to find the first multiple of 13 in a list of numbers:

iex> try do
...>   Enum.each -50..50, fn(x) ->
...>     if rem(x, 13) == 0, do: throw(x)
...>   end
...>   "Got nothing"
...> catch
...>   x -> "Got #{x}"
...> end
"Got -39"

Since Enum does provide a proper API, in practice Enum.find/2 is the way to go:

iex> Enum.find -50..50, &(rem(&1, 13) == 0)
-39

Exits

All Elixir code runs inside processes that communicate with each other. When a process dies of “natural causes” (e.g., unhandled exceptions), it sends an exit signal. A process can also die by explicitly sending an exit signal:

iex> spawn_link fn -> exit(1) end
#PID<0.56.0>
** (EXIT from #PID<0.56.0>) 1

exit can also be “caught” using try/catch:

iex> try do
...>   exit "I am exiting"
...> catch
...>   :exit, _ -> "not really"
...> end
"not really"

Using try/catch is already uncommon and using it to catch exits is even more rare.

After

Sometimes it’s necessary to ensure that a resource is cleaned up after some action that could potentially raise an error. The try/after construct allows you to do that. For example, we can open a file and use an after clause to close it–even if something goes wrong:

iex> {:ok, file} = File.open "sample", [:utf8, :write]
iex> try do
...>   IO.write file, "olá"
...>   raise "oops, something went wrong"
...> after
...>   File.close(file)
...> end
** (RuntimeError) oops, something went wrong

Sometimes you may want to wrap the entire body of a function in a try construct, often to guarantee some code will be executed afterwards. In such cases, Elixir allows you to omit the try line:

iex> defmodule RunAfter do
...>   def without_even_trying do
...>     raise "oops"
...>   after
...>     IO.puts "cleaning up!"
...>   end
...> end
iex> RunAfter.without_even_trying
cleaning up!
** (RuntimeError) oops

Typespecs and behaviours


Types and specs

It’s used for:

  • declaring custom data types;
  • declaring typed function signatures (specifications).

Function specifications

@spec round(number) :: integer
def round(number), do: # implementation...

Defining custom types

defmodule LousyCalculator do
  @typedoc """
  Just a number followed by a string.
  """
  @type number_with_remark :: {number, String.t}

  @spec add(number, number) :: number_with_remark
  def add(x, y), do: {x + y, "You need a calculator to do that?"}

  @spec multiply(number, number) :: number_with_remark
  def multiply(x, y), do: {x * y, "It is like addition on steroids."}
end

Custom types defined through @type are exported and available outside the module they’re defined in:

defmodule QuietCalculator do
  @spec add(number, number) :: number
  def add(x, y), do: make_quiet(LousyCalculator.add(x, y))

  @spec make_quiet(LousyCalculator.number_with_remark) :: number
  defp make_quiet({num, _remark}), do: num
end

If you want to keep a custom type private, you can use the @typep directive instead of @type.

Static code analysis

Typespecs are not only useful to developers and as additional documentation. The Erlang tool Dialyzer, for example, uses typespecs in order to perform static analysis of code. That’s why, in the QuietCalculator example, we wrote a spec for the make_quiet/1 function even if it was defined as a private function.

Behaviours

Behaviours provide a way to:

  • Define a set of functions that have to be implemented by a module;
  • Ensure that a module implements all the functions in that set.

Defining behaviours

defmodule Parser do
  @callback parse(String.t) :: any
  @callback extensions() :: [String.t]
end

Modules adopting the Parser behaviour will have to implement all the functions defined with the @callback directive. As you can see, @callback expects a function name but also a function specification like the ones used with the @spec directive we saw above.

Adopting a behaviour is straightforward:

defmodule JSONParser do
  @behaviour Parser

  def parse(str), do: # ... parse JSON
  def extensions, do: ["json"]
end
defmodule YAMLParser do
  @behaviour Parser

  def parse(str), do: # ... parse YAML
  def extensions, do: ["yml"]
end

Erlang libraries


As you grow more proficient in Elixir, you may want to explore the Erlang STDLIB Reference Manual in more detail. Check below some of the most widely used libraries.

The binary module

The built-in Elixir String module handles binaries that are UTF-8 encoded. The binary module is useful when you are dealing with binary data that is not necessarily UTF-8 encoded.

Formatted text output

Elixir does not contain a function similar to printf found in C and other languages. Luckily, the Erlang standard library functions :io.format/2 and :io_lib.format/2 may be used. The first formats to terminal output, while the second formats to an iolist. The format specifiers differ from printf, refer to the Erlang documentation for details.

The crypto module

The crypto module contains hashing functions, digital signatures, encryption and more:

iex> Base.encode16(:crypto.hash(:sha256, "Elixir"))
"3315715A7A3AD57428298676C5AE465DADA38D951BDFAC9348A8A31E9C7401CB"

The :crypto module is not part of the Erlang standard library, but is included with the Erlang distribution. This means you must list :crypto in your project’s applications list whenever you use it. To do this, edit your mix.exs file to include.

The digraph module

The digraph module (as well as digraph_utils) contains functions for dealing with directed graphs built of vertices and edges. After constructing the graph, the algorithms in there will help finding for instance the shortest path between two vertices, or loops in the graph.

Erlang Term Storage

The modules ets and dets handle storage of large data structures in memory or on disk respectively.

ETS lets you create a table containing tuples. By default, ETS tables are protected, which means only the owner process may write to the table but any other process can read. ETS has some functionality to be used as a simple database, a key-value store or as a cache mechanism.

The functions in the ets module will modify the state of the table as a side-effect.

iex> table = :ets.new(:ets_test, [])
# Store as tuples with {name, population}
iex> :ets.insert(table, {"China", 1_374_000_000})
iex> :ets.insert(table, {"India", 1_284_000_000})
iex> :ets.insert(table, {"USA", 322_000_000})
iex> :ets.i(table)
<1   > {"USA", 322000000}
<2   > {"China", 1_374_000_000}
<3   > {"India", 1_284_000_000}

The math module

The math module contains common mathematical operations covering trigonometry, exponential and logarithmic functions.

The queue module

The queue is a data structure that implements (double-ended) FIFO (first-in first-out) queues efficiently:

iex> q = :queue.new
iex> q = :queue.in("A", q)
iex> q = :queue.in("B", q)
iex> {value, q} = :queue.out(q)
iex> value
{:value, "A"}
iex> {value, q} = :queue.out(q)
iex> value
{:value, "B"}
iex> {value, q} = :queue.out(q)
iex> value
:empty

The rand module

rand has functions for returning random values and setting the random seed.

iex> :rand.uniform()
0.8175669086010815
iex> _ = :rand.seed(:exs1024, {123, 123534, 345345})
iex> :rand.uniform()
0.5820506340260994
iex> :rand.uniform(6)
6

The zip and zlib modules

The zip module lets you read and write zip files to and from disk or memory, as well as extracting file information.

References


  • http://elixir-lang.org/getting-started/introduction.html/
  • Cover picture: The Banquet in the Pine Forest (1482/3) is the third painting in Sandro Botticelli’s series The Story of Nastagio degli Onesti, which illustrates events from the Eighth Story of the Fifth Day. (Public Domain)