Distributed portals with Elixir

Notebook dependencies and setup
Introduction

2 cells collapsed

Basic data structures

7 cells collapsed

Pattern matching

22 cells collapsed

Modeling portal doors with Agents

26 cells collapsed

Portal transfers

21 cells collapsed

Inspecting portals with Protocols

9 cells collapsed

Distributed transfers

22 cells collapsed

Wrapping up

1 cell collapsed

Export

Here you can preview and directly export the notebook source.
Live Markdown IEx session
distributed_portals_with_elixir.exs
# Run as: iex --dot-iex path/to/notebook.exs

# Title: Distributed portals with Elixir

# ── Introduction ──

# This notebook is a fast-paced introduction to the Elixir
# programming language. We will explore both basic and advanced
# concepts to implement our own version of [the Portal
# game](http://en.wikipedia.org/wiki/Portal_(video_game)) to
# transfer data across notebooks using Elixir's distribution
# capabilities.

# For a more structured introduction to the language, see [Elixir's
# Getting Started guide](https://elixir-lang.org/getting-started/introduction.html)
# and [the many learning resources available](https://elixir-lang.org/learning.html).

# ### The plan ahead

# The Portal game consists of a series of puzzles that must be
# solved by teleporting the player's character and simple objects
# from one place to another.

# In order to teleport, the player uses the Portal gun to shoot doors
# onto flat planes, like a floor or a wall. Entering one of those doors
# teleports you to the other:

# ![](files/portal-drop.jpeg)

# Our version of the Portal game will use Elixir to shoot doors of
# different colors and transfer data between them! We will even learn how
# we can distribute doors across different machines in our network:

# ![](files/portal-list.jpeg)

# Here is what we will learn:

# * Elixir's basic data structures
# * Pattern matching
# * Using agents for state
# * Using structs for custom data structures
# * Extending the language with protocols
# * Distributed Elixir nodes

# At the end of this notebook, we will make the following code work:

# ```elixir
# # Shoot two doors: one orange, another blue
# Portal.shoot(:orange)
# Portal.shoot(:blue)

# # Start transferring the list [1, 2, 3, 4] from orange to blue
# portal = Portal.transfer(:orange, :blue, [1, 2, 3, 4])

# # This will output:
# #
# #   #Portal<
# #          :orange <=> :blue
# #     [1, 2, 3, 4] <=> []
# #   >

# # Now every time we call push_right, data goes to blue
# Portal.push_right(portal)

# # This will output:
# #
# #   #Portal<
# #          :orange <=> :blue
# #        [1, 2, 3] <=> [4]
# #   >
# ```

# Intrigued? Let's get started!

# ── Basic data structures ──

# Elixir has numbers, strings, and variables. Code comments start with `#`:

# Numbers
IO.inspect(40 + 2)

# Strings
variable = "hello" <> " world"
IO.inspect(variable)

# Executing the cell above prints the number `42` and the string
# `"hello world"`. To do so, we called the function `inspect` in
# the `IO` module, using the `IO.inspect(...)` notation. This function
# prints the given data structure to your terminal - in this case,
# our notebook - and returns the value given to it.

# Elixir also has three special values, `true`, `false`, and `nil`.
# Everything in Elixir is considered to be a truthy value, except for
# `false` and `nil`:

# && is the logical and operator
IO.inspect(true && true)
IO.inspect(13 && 42)

# || is the logical or operator
IO.inspect(true || false)
IO.inspect(nil || 42)

# For working with collections of data, Elixir has three data types:

# Lists (typically hold a dynamic amount of items)
IO.inspect([1, 2, "three"])

# Tuples (typically hold a fixed amount of items)
IO.inspect({:ok, "value"})

# Maps (key-value data structures)
IO.inspect(%{"key" => "value"})

# In the snippet above, we also used a new data structure represented
# as `:ok`. All values starting with a leading `:` in Elixir are called
# **atoms**. Atoms are used as identifiers across the language. Common
# atoms are `:ok` and `:error`. Which brings us to the next topic: pattern
# matching.

# ── Pattern matching ──

# The `=` operator in Elixir is a bit different from the ones we see
# in other languages:

x = 1
x

# So far so good, but what happens if we invert the operands?

1 = x

# It worked! That's because Elixir tries to match the right side
# against the left side. Since both are set to `1`, it works. Let's
# try something else:

2 = x

# Now the sides did not match, so we got an error. We use pattern
# matching in Elixir to match on collection too. For example, we
# can use `[head | tail]` to extract the head (the first element)
# and tail (the remaining ones) from a list:

[head | tail] = [1, 2, 3]
IO.inspect(head)
IO.inspect(tail)

# Matching an empty list against `[head | tail]` causes a match error:

[head | tail] = []

# Finally, we can also use the `[head | tail]` expression to add
# elements to the head of a list:

list = [1, 2, 3]
[0 | list]

# We can also pattern match on tuples. This is often used to match
# on the return types of function calls. For example, take the function
# `Date.from_iso8601(string)`, which returns `{:ok, date}` if the
# string represents a valid date, in the format `YYYY-MM-DD`, otherwise
# it returns `{:error, reason}`:

# A valid date
Date.from_iso8601("2020-02-29")

# An invalid date
Date.from_iso8601("2020-02-30")

# Now, what happens if we want our code to behave differently depending
# if the date is valid or not? We can use `case` to pattern match on
# the different tuples:

# Give an invalid date as input
input = "2020-02-30"

# And then match on the return value
case Date.from_iso8601(input) do
  {:ok, date} ->
    "We got a valid date: #{inspect(date)}"

  {:error, reason} ->
    "Oh no, the date is invalid. Reason: #{inspect(reason)}"
end

# In this example, we are using `case` to pattern match on the different
# outcomes of the `Date.from_iso8601`. We say the `case` above has two
# clauses, one matching on `{:ok, date}` and another on `{:error, reason}`.
# Now try changing the `input` variable above and reevaluate the cell
# accordingly. What happens when you give it a valid date?

# Finally, we can also pattern match on maps. This is used to extract the
# values for the given keys:

map = %{:elixir => :functional, :python => :object_oriented}
%{:elixir => type} = map
type

# If the key does not exist on the map, it raises:

%{:c => type} = map

# With pattern matching out of the way, we are ready to start our Portal
# implementation!

# ── Modeling portal doors with Agents ──

# Elixir data structures are immutable. In the examples above, we never
# mutated the list. We can break a list apart or add new elements to the
# head, but the original list is never modified.

# That said, when we need to keep some sort of state, like the data
# transferring through a portal, we must use an abstraction that stores
# this state for us. One such abstraction in Elixir is called an agent.
# Before we use agents, we need to briefly talk about anonymous functions.

# Anonymous functions are a mechanism to represent pieces of code that we
# can pass around and execute later on:

adder = fn a, b -> a + b end

# An anonymous function is delimited by the words `fn` and `end` and an
# arrow `->` is used to separate the arguments from the anonymous function
# body. We can now call the anonymous function above multiple times by
# providing two numbers as inputs:

adder.(1, 2)

adder.(3, 5)

# In Elixir, we also use anonymous functions to initialize, get, and update
# the agent state:

{:ok, agent} = Agent.start_link(fn -> [] end)

# In the example above, we created a new agent, passing a function that
# returns the initial state of an empty list. The agent returns
# `{:ok, #PID<...>}`, where PID stands for a process identifier, which
# uniquely identifies the agent. Elixir has many abstractions for concurrency,
# such as agents, tasks, generic servers, but at the end of the day they all
# boil down to processes. When we say processes in Elixir, we don't mean
# Operating System processes, but rather Elixir Processes, which are lightweight
# and isolated, allowing us to run hundreds of thousands of them on the same
# machine.

# We store the agent's PID in the `agent` variable, which allows us to
# send messages to get the agent's state:

Agent.get(agent, fn list -> list end)

# As well as update it before reading again:

Agent.update(agent, fn list -> [0 | list] end)
Agent.get(agent, fn list -> list end)

# We will use agents to implement our portal doors.

# Whenever we need to encapsulate logic in Elixir, we create modules,
# which are essentially a collection of functions. We define modules
# with `defmodule` and functions with `def`. Our functions will
# encapsulate the logic to interact with the agent, using the API we
# learned in the cells above:

defmodule Portal.Door do
  use Agent

  def start_link(color) when is_atom(color) do
    Agent.start_link(fn -> [] end, name: color)
  end

  def get(door) do
    Agent.get(door, fn list -> list end)
  end

  def push(door, value) do
    Agent.update(door, fn list -> [value | list] end)
  end

  def pop(door) do
    Agent.get_and_update(door, fn list ->
      case list do
        [h | t] -> {{:ok, h}, t}
        [] -> {:error, []}
      end
    end)
  end

  def stop(door) do
    Agent.stop(door)
  end
end

# We declare a module by giving it a name, in this case, `Portal.Door`.
# At the top of the module, we say `use Agent`, which brings some
# `Agent`-related functionality into the module.

# The first function is `start_link`, which we often refer to as
# `start_link/1`, where the number 1 is called the "arity" of the
# function and it indicates the number of arguments it receives.
# Then we check that the argument is an atom and proceed to call
# `Agent.start_link/2`, as we did earlier in this section, except we are
# now passing `name: color` as an argument. By giving a name to the
# Agent, we can refer to it anywhere by its name, instead of using its
# PID.

# The next two functions, `get/1` and `push/2` perform simple operation
# to the agent, reading its state and adding a new element respectively.
# Let's take a look at them:

Portal.Door.start_link(:pink)
Portal.Door.get(:pink)

# Note how we didn't need to store the PID anywhere and we can use the
# atom `:pink` to refer to the door and read its state. If the door
# already exists, and we try to start another one with the same name,
# it returns an `{:error, reason}` tuple instead of `{:ok, pid}`:

Portal.Door.start_link(:pink)

# Next, let's push some events:

Portal.Door.push(:pink, 1)
Portal.Door.push(:pink, 2)
Portal.Door.get(:pink)

# We pushed some events and they show up in our state. Although, note
# they appear in reverse order. That's because we are always adding
# new entries to the top of the list.

# The fourth function we defined is called `pop/1`. If there is any
# item in the agent, it takes the head of the list and returns it
# wrapped in a `{:ok, value}` tuple. However, if the list is empty,
# it returns `:error`.

IO.inspect(Portal.Door.pop(:pink))
IO.inspect(Portal.Door.pop(:pink))
IO.inspect(Portal.Door.pop(:pink))

Portal.Door.get(:pink)

# Finally, the last function, `stop/1`, simply terminates the agent,
# effectively closing the door. Let's try it:

Portal.Door.stop(:pink)

# Now, if we try to do anything with it, it will raise:

Portal.Door.get(:pink)

# Note the error message points out why the operation did not work, great!

# ── Portal transfers ──

# Our portal doors are ready so it is time to start working on portal
# transfers! In order to store the portal data, we are going to create a
# struct named `Portal`. Let's first learn what structs are about.

# Structs define data structures with pre-defined keys. The keys are verified
# at compilation time, so if you make a typo in the key name, you get an error
# early on. Structs are defined inside modules, by calling the `defstruct` with
# a list of atom keys. Let's define a `User` struct with the fields `:name`
# and `:age`:

defmodule User do
  defstruct [:name, :age]
end

# Now, we can create structs using the `%User{...}` notation:

user = %User{name: "john doe", age: 27}

# We can access struct fields using the `struct.field` syntax:

user.name

# We can pattern match on structs too:

%User{age: age} = user
age

# Finally, let's see what happens if we do a typo in a field:

%User{agee: age} = user

# Now we are ready to define our `Portal` struct. It will have
# two fields, `:left` and `:right`, which point respectively to
# the portal door on the left and the door on the right. Our goal
# is to transfer data from the left door to the right one. The
# `Portal` module, where we define our struct, will also have
# four other functions:

# * `shoot(color)` - shoots a door of the given color. This is
#   a wrapper around `Portal.Door.start_link/1`

# * `transfer(left_door, right_door, data)` - starts a transfer
#   by loading the given `data` to `left_door` and returns a
#   `Portal` struct

# * `push_right(portal)` - receives a portal and continues the
#   transfer by pushing data from the left to the right

# * `close(portal)` - closes the portal by explicitly stopping
#   both doors

# Let's implement them:

defmodule Portal do
  defstruct [:left, :right]

  def shoot(color) do
    Portal.Door.start_link(color)
  end

  def transfer(left_door, right_door, data) do
    # First add all data to the portal on the left
    for item <- data do
      Portal.Door.push(left_door, item)
    end

    # Returns a portal struct with the doors
    %Portal{left: left_door, right: right_door}
  end

  def push_right(portal) do
    # See if we can pop data from left. If so, push the
    # popped data to the right. Otherwise, do nothing.
    case Portal.Door.pop(portal.left) do
      :error   -> :ok
      {:ok, h} -> Portal.Door.push(portal.right, h)
    end

    # Let's return the portal itself
    portal
  end

  def close(portal) do
    Portal.Door.stop(portal.left)
    Portal.Door.stop(portal.right)
    :ok
  end
end

# The `Portal` modules defines a struct followed by a `shoot/1` function.
# The function is just a wrapper around `Portal.Door.start_link/1`. Then
# we define the `transfer/3` function, which loads the given data into the
# left door and returns a Portal struct. Finally, `push_right/1` gets data
# from the door on the left and puts it on the right door. Let's give it a try:

Portal.shoot(:orange)
Portal.shoot(:blue)
portal = Portal.transfer(:orange, :blue, [1, 2, 3])

# The above returns the `%Portal{}` struct. We can check the data has been
# loaded into the left door:

Portal.Door.get(:orange)

# Note the list is reversed - and we knew that! - as we always add items
# on the top. But we will use that to our advantage soon. Let's start pushing
# data to the right:

Portal.push_right(portal)
Portal.Door.get(:blue)

# Since the list is reversed, we can see that we pushed the number `3`
# to the right, which is exactly what we expected. If you reevaluate the
# cell above, you will see data moving to the right, as our portal doors
# are stateful.

# Our portal transfer seems to work as expected! Now let's clean up and
# close the transfer:

Portal.close(portal)

# We have made some good progress in our implementation, so now let's work
# a bit on the presentation. Currently, the Portal is printed as a struct:
# `%Portal{left: :orange, right: :blue}`. It would be nice if we actually
# had a printed representation of the portal transfer, allowing us to see
# the portal processes as we push data.

# ── Inspecting portals with Protocols ──

# We already know that Elixir data structures data can be printed by Livebook.
# After all, when we type `1 + 2`, we get `3` back. However, can we customize
# how our own data structures are printed?

# Yes, we can! Elixir provides protocols, which allows behaviour to be extended
# and implemented for any data type, like our `Portal` struct, at any time.

# For example, every time something is printed in Livebook, or in Elixir's
# terminal, Elixir uses the `Inspect` protocol. Since protocols can be extended
# at any time, by any data type, it means we can implement it for `Portal` too.
# We do so by calling `defimpl/2`, passing the protocol name and the data
# structure we want to implement the protocol for. Let's do it:

defimpl Inspect, for: Portal do
  def inspect(%Portal{left: left, right: right}, _) do
    left_door  = inspect(left)
    right_door = inspect(right)

    left_data  = inspect(Enum.reverse(Portal.Door.get(left)))
    right_data = inspect(Portal.Door.get(right))

    max = max(String.length(left_door), String.length(left_data))

    """
    #Portal<
      #{String.pad_leading(left_door, max)} <=> #{right_door}
      #{String.pad_leading(left_data, max)} <=> #{right_data}
    >\
    """
  end
end

# In the snippet above, we have implemented the `Inspect` protocol for the
# `Portal` struct. The protocol expects one function named `inspect` to be
# implemented. The function expects two arguments, the first is the `Portal`
# struct itself and the second is a set of options, which we don't care about
# for now.

# Then we call `inspect` multiple times, to get a text representation of both
# `left` and `right` doors, as well as to get a representation of the data
# inside the doors. Finally, we return a string containing the portal presentation
# properly aligned.

# That's all we need! Let's start a new transfer and see how it goes:

Portal.shoot(:red)
Portal.shoot(:purple)
portal = Portal.transfer(:red, :purple, [1, 2, 3])

# Sweet! Look how Livebook automatically picked up the new representation.
# Now feel free to call `push_right` and see what happens:

Portal.push_right(portal)

# Feel free to reevaluate the cell above a couple times. Once you are done,
# run the cell below to clean it all up and close the portal:

Portal.close(portal)

# There is just one topic left...

# ── Distributed transfers ──

# With our portals working, we are ready to give distributed transfers a try.
# However, before we start, there is one big disclaimer:

# > The feature we are going to implement will allow us to share data across
# > two separate notebooks using the Erlang Distribution. This section is a great
# > experiment to understand how things work behind the scenes but **sharing data
# > across notebooks as done here is a bad idea in practice**. If this topic
# > interests you, we recommend picking up one of the many available learning
# > resources available for Elixir, which will provide a more solid ground to
# > leverage the Erlang VM and its distributed features.

# ### Distribution 101

# When Livebook executes the code in a notebook, it starts a separate Elixir
# runtime to do so. Since Livebook itself is also written in Elixir, it uses
# the Erlang Distribution to communicate with this Elixir runtime. We can
# get the name of the Elixir node our notebook is running on like this:

node()

# By executing the code above, we can see node names are atoms. By default,
# Livebook only connects to nodes running on the same machine, but you can
# also configure it to connect to runtimes across machines.

# We can also get a list of all nodes our runtime is connected to by calling
# `Node.list()`, let's give it a try:

Node.list()

# If you execute the code above, you get... an empty list!?

# The reason why we get an empty list is because, by default, Erlang Distribution
# is a fully mesh. This means all nodes can see all nodes in the network. However,
# because we want the notebook runtimes to be isolated from each other, we start
# each runtime as a _hidden_ node. We can ask Elixir to give us all hidden nodes
# instead:

Node.list(:hidden)

# Much better! We see one node, which is the Livebook server itself.

# Now there is one last piece of the puzzle: in order for nodes to connect to each
# other, they need to have the same cookie. The cookie serves as a very simple
# authentication mechanism. We can read the cookie of our current notebook runtime
# like this:

Node.get_cookie()

# Now we have everything we need to connect across notebooks.

# ### Notebook connections

# In order to connect across notebooks, open up [a new empty notebook](/new)
# in a separate tab, copy and paste the code below to this new notebook,
# and execute it:

# ```elixir
# IO.inspect node()
# IO.inspect Node.get_cookie()
# :ok
# ```

# Now paste the result of the other node name and its cookie in the variables below:

other_node = :"name-of-the@other-node"
other_cookie = :"value-of-the-other-cookie"

# With both variables defined, let's connect to the other node with the given cookie:

Node.set_cookie(other_node, other_cookie)
Node.connect(other_node)

# If it returns true, it means it connected as expected and we are ready to start a
# distributed transfer. The first step is to go to the other notebook and shoot a door.
# You can try calling the following there:

# ```elixir
# Portal.shoot(:blue)
# ```

# However, if you try the above, it will fail! This happens because the
# Portal code has been defined in this notebook but it is not available
# in the other notebook. If we were working on an actual Elixir project,
# this issue wouldn't exist, because we would start multiple nodes on top
# of the same codebase with the same modules, but we can't do so here.
# To work around this, copy the cell that defines the `Portal.Door` module
# from this notebook into the other notebook and execute it. Now in the other
# node you should be able to start a door:

# ```elixir
# Portal.Door.start_link(:blue)
# ```

# ### Cross-node references

# Now that we have spawned a door on the other notebook, we can directly read
# its content from this notebook. So far, we have been using atoms to represent
# doors, such as `:blue`, but we can also use the `{name, node}` notation to refer
# to a process in another node. Let's give it a try:

blue = {:blue, other_node}
Portal.Door.get(blue)

# It works! We could successfully read something from the other node. Now,
# let's shoot a yellow portal on this node and start a transfer between them:

Portal.shoot(:yellow)
yellow = {:yellow, node()}
portal = Portal.transfer(yellow, blue, [1, 2, 3, 4])

# Our distributed transfer was started. Now let's push right:

Portal.push_right(portal)

# If you go back to the other notebook and run `Portal.Door.get(:blue)`,
# you should see it has been updated with entries from this notebook!
# The best part of all is that we enabled distributed transfers without
# changing a single line of code!

# Our distributed portal transfer works because the doors are just processes
# and accessing/pushing the data through doors is done by sending messages
# to those processes via the Agent API. We say sending a message in Elixir
# is location transparent: we can send messages to any PID regardless if it
# is in the same node as the sender or in different nodes of the same network.

# ── Wrapping up ──

# So we have reached the end of this notebook with a fast paced introduction
# to Elixir! It was a fun ride and we went from manually starting doors to
# distributed portal transfers.

# To learn more about Elixir, we welcome you to explore our [website](http://elixir-lang.org),
# [read our Getting Started guide](https://elixir-lang.org/getting-started/introduction.html),
# and [many of the available learning resources](https://elixir-lang.org/learning.html).

# Finally, huge thanks to Augie De Blieck Jr. for the drawings in this tutorial.

# See you around!