Recent Posts

Local Clusters with epmd

Posted on 26 Mar 2019

For Grapevine, I recently started a single machine cluster. There are two nodes, one for the main application (sockets and Phoenix) and one as a base for telnet connections.

With this I can redeploy the main application and not disconnect players from their game sessions. This is very important as telnet connections cannot drop if they are open.

Clustering

The start of this cluster was incredibly simple. I installed libcluster and configured each node to connect to the other via named nodes. Each node was simply their release name @127.0.0.1 by default thanks to distillery's default configuration.

I started each node and all was well. They both connected and I was off to the races. Until I wasn't.

Issues

This worked well until I went to deploy. I was never able to successfully deploy each node separately without one of the two not being able to connect to the other. It was very reliable which node couldn't connect. The second node to start could never connect to the other node.

I tried a few ways of getting around this but couldn't figure out what was wrong. I scratched my head for an evening before letting it sit overnight.

Starting epmd Separately

What I think was the issue, was that epmd was being started automatically by the first node to boot. The way I was deploying was simply overwriting the entire release directory with the new code. I think this screwed something up, I'm not 100% sure about this though.

By starting epmd as a separate service, the issues stopped. I set up my service files to require epmd before starting. In a sense I pushed the issue up the supervision tree!

Service Files

This is the epmd service that Grapevine uses. Ansible places the epmd daemon at /home/deploy/epmd (in this file.)

[Unit]
Description=Erlang Port Mapper Daemon
After=network.target

[Service]
User=deploy
Group=deploy
WorkingDirectory=/home/deploy/
Environment=LANG=en_US.UTF-8
ExecStart=/home/deploy/epmd
SyslogIdentifier=epmd
RemainAfterExit=no
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

And the grapevine.service requires that it is booted with this line:

[Install]
WantedBy=epmd.service

Conclusion

If you have a local cluser, you should avoid node connection issues by starting epmd separately and not allowing for the automatic daemon start. As an extra bonus, you can now use the epmd binary to communicate easily with the running daemon. This lets you know each node is alive and connected, use epmd -names for this.

Finally, all of this code is open source so you can inspect what Grapevine does for clustering. See the main repo here and the telnet node repo here.

See more about epmd.

ExVenture & Grapevine Updates for February 2019

Posted on 28 Feb 2019

The last month of ExVenture and Grapevine saw some refactors in ExVenture and a brand new web client in Grapevine.

Links for ExVenture & Grapevine:

Grapevine Web Client

Grapevine Web Client

The biggest new feature of Grapevine, and the thing that has consumed most of my time is a new web client for accessing telnet based games. This feature is turned on by the game administrator and configured by them for all users. This way a consistent and simple interface is given to everyone, keeping configuration on the player non-existant at the moment.

Game admins can configure only what gauges should show at the bottom of the client right now. I'm slowly working on more options, but this is a good start. It's powered by GMCP, a telnet protocol extension.

The client is powered by React and Redux, and even has a few reducer tests. This is my first go around with a "real" react and redux application and I'm liking it. It finally "clicked" as to why I'd want to add in redux this time.

I even set it up so games can have a subdomain on their own domain that launches directly to the web client. If you have a game and want to enable this, let me know!

Grapevine Clustering

In order to enable the web client to not drop connections on a deploy, I pulled a node that holds the connection to games out of the main node. This hosts just the telnet connections and that's about it.

This lets me deploy the main Grapevine code as often as I like and not have it drop connections. Users still see a brief period of web client <-> Grapevine disconnection but they eventually reconnect and pick right back up.

I need to work on better messaging around this, but the core works well.

Check out the new telnet only application.

ExVenture Character Tuple

Not to forget that ExVenture exists, I started work on a fairly large refactor that tackles something that's been annoying me for a while. The character target system passed around tagged tuples that tag what kind of character the tuple is, e.g. {:player, player}.

This is really ugly to read in the code so I've been working through the code base removing those in favor of pushing around the plain Character.Simple struct, adding in a type field when its necessary to distinguish between the two. I'm hoping that this will also let me collapse the Room process caring about the two as separate entities.

See the current state of this pull request on GitHub.

Social Updates

We have a few new patrons this month, welcome! Thanks for supporting both ExVenture and Grapevine. If you would like to support both projects, check out the Patreon.

I will be at Lonestar Elixir at the end of this week, giving a talk about monitoring your application with Prometheus.

I have also continued streaming over on Twitch every Monday at 12 PM EST doing ExVenture or Grapevine development. Join me over at https://www.twitch.tv/smartlogictv.

Finally, SmartLogic launched a new podcast centered around Elixir in Production. Find it over at podcast.smartlogic.io

Next Month

Next month I will continue expanding the features that the web client can do. I feel like it's pretty close to being the right amount of simple and usable. I don't want to provide that many options, as MUD clients that are extremely powerful already exist. I also want to get back to ExVenture and keep that pushing forward, but we'll see where the wind takes me.

ExVenture Updates for January 2019

Posted on 25 Jan 2019

The last month of ExVenture saw a continuation of refactors and some exciting changes in Gossip & Grapevine.

Links for MidMUD & ExVenture:

Gossip

Gossip got a few nice improvements. I added the ability to scrape telnet based games that aren't hooked up to the chat network. There is a protocol called MUD Server Stats Protocol (MSSP) that probes telnet connections for game information. Gossip now sweeps every hour for compatible games, adding to the list of online games.

Achievements are also on the move. You can add achievements to your game and have them display. I haven't been working on unlocking them yet, but the core is there. I was delayed on unlocking them because of the weird split in Grapevine and Gossip.

Grapevine

Speaking of that, Gossip and Grapevine have merged into just Grapevine. I added all of the features to Gossip, and then renamed Gossip to Grapevine. This split was sort of awkward and made it confusing for people signing up. It was also harder than it needed to be to sync new stuff from Gossip to Grapevine. With this in place it should be much easier to add new features.

Continuing with that, I've been working on the design for Grapevine. There's a new home page that displays online games with their new cover art. This makes it look more familiar to other gaming stores, even if Grapevine isn't a store.

Proficiencies

I've also been working on some new game features. I've been working on proficiencies which is a way of gating things behind a rank. An example might be a Swimming proficiency, which you can require 5 ranks in in order to move between two rooms. If you have 4 ranks or under, you are prevented from moving through. Players can hone ranks by spending experience points.

I think this is a nice simple mechanic that can be used in interesting ways. No proficiencies are bundled into ExVenture, you only have the ones you design for your game. You can watch me work on honing proficiencies on this YouTube video.

CI & Test refactors

I worked on setting up CI again for ExVenture (and Grapevine.) As part of this I worked a lot on refactoring the test suite to remove global state Agents. Instead I started sending messages to the test process as code interacts with remote processes. It was pretty nice to get this working and also dramatically stabalized the test runs.

Also as part of CI, I'm building a new release inside of Docker and pushing to S3. I don't have a nice URL for these yet, but the top commit on the master branch should have a built release you can use to deploy. I want to figure out how to generate a clean URL for easy use. I would like to do a blog about how this works come next month.

Small Tweaks

  • Player URL is case insenitive on grapevine
  • Display channels that a game is connected to on gossip
  • ExVenture (un)subscribes to Gossip channel updates
  • Lots of new metrics
  • Refactor Game.Help
  • Continuing migrating formatting calls to the new template system
  • Wrap calls to injected modules (as module attributes) in a single module
  • Help updates across the cluster
  • Password resets for Gossip accounts
  • Work continues on the react web client

Social Updates

I'll be at Lonestar Elixir at the end of next month talking about getting metrics from your application with Prometheus. I hope to see you there!

I've been streaming over on Twitch every Monday at 12 PM EST doing ExVenture or Grapevine development. Join me over at https://www.twitch.tv/smartlogictv.

Next Month

Next Month I would like to continue expanding what proficiencies can influence. I think it would be cool to get them in front of items and a few other things. I will also keep pushing forward on the new Grapevine. I'm looking at adding a fairly simple web client to Grapevine to easily connect to telnet games, among other things.

ExVenture Updates for December 2018

Posted on 29 Dec 2018

The last month of ExVenture saw a lot of internal refactors and deployment updates.

Links for MidMUD & ExVenture:

NPC Events

Events got a huge refactor to consolidate a lot of duplicate code around actions. Previously it was not straight forward which events could have what actions, and each event implemented the action separately. This refactor sets up events and actions as full structs and modules.

Now each event lists what actions it allows. Each action is implemented once. All events also can have multiple actions in order to have more realistic NPCs.

Check out the documentation for these events.

Pull Request.

Web Client in React

Lorecrafting in the discord has started on a major refactor of the web client, switching over to react. It's still fairly in progress, but is moving along nicely.

Venture Markup Language

In order to help the web client and more solidify the internal markup format, the Venture Markup Language (VML) was named. Nothing much changes other than giving it a name and actually doing parsing instead of regex replace.

There is now a set of leex and yecc parsing modules. VML now has some documentation.

Pull Request

Deployment

ExVenture isn't the easiest thing to deploy in production at the moment, so I finally got around to working on making that easier. As part of this I finally got around to writing ansible scripts and a small deployment script to help everything out.

See more on deploying ExVenture in the last post.

ExVenture World

Continuing with this, I have started a new hosting service called ExVenture World. This uses the ansible scripts and terraform to create new game instances in Digital Ocean very quickly. I plan on leaving this not fully automated for a while and eventually automate it fully if more people are interested in hosted games.

If you are interested in a hosted version, please let me know over on the discord channel.

Small Tweaks

  • Distillery 2.0 config provider, for everything
  • Gossip design tweaks
  • Use systemd
  • Migrate to new servers
  • Builder role
  • Grapevine only login

Social Updates

A few times people have requested a forum to be set up for ExVenture, this now exists over at forums.exventure.org.

We have a few new patrons on the Patron, thanks for supporting!

I will be at Lonestar Elixir 2019 showing off adding prometheus metrics to your application, and Gossip will be my demo app. If you're there, make sure to say hello!

Next Month

For the next month, I'm working on some new game mechanics, the first of which is proficiencies. I am hoping to get back around to some of the other mechanics that are already in and expand them a bit, such as items. I would like to start moving on achievements for Gossip as well.

Deploying ExVenture

Posted on 15 Dec 2018

Up until recently, how to deploy ExVenture has been a bit of a mystery. I have done it a few times, for MidMUD most notably, but it was fairly tricky to do and entirely done by hand.

I also relied on some parts of linux like /etc/profile.d/ to ensure environment variables were loaded, so starting the application could only be done by a login shell. It wasn't the best, but hey, it worked.

This week however, a few more people were starting to get interested in ExVenture and asking about how to host it and I finally got embarassed enough to get something more formal around how to deploy it.

I also started a hosting service called ExVenture World if anyone is interested in having a game without having to deploy it. See the announcement post for ExVenture World on Patreon.

Vagrant

Vagrant is a tool for spinning up local VMs quickly, and being able to quickly recreate them from a base box. We're going to use this to get a local install going. Install Vagrant before continuing.

vagrant up

After a few minutes, you'll have a local machine you can ssh into. Vagrant will provision python for us on first boot so that we can run ansible on it in our next step.

Something I've tended to have issues with on Vagrant is SSH known host issues on recreating a VM. I just learned about this snippet you can place in ~/.ssh/config to ignore host checks. Do make sure it's only enabled for 127.0.0.1 as this is very insecure.

Host 127.0.0.1
  HostName 127.0.0.1
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  IdentitiesOnly yes

Ansible

Ansible is a tool for configuring remote servers. They run playbooks to make sure a server is configured the same each time. See more information about it and how to install it on their site.

First install the roles you'll be using to provision the server. This uses Ansible Galaxy to pull down remote roles.

ansible-galaxy install -r deploy/requirements.yml

Next, run the setup playbook on the local machine. This will configure the local VM to be provisioned for the next few steps. Make sure to update this line in the deploy/group_vars/all.yml line to include your GitHub user. It will download any keys you have associated to your account and let you into the deploy user.

ansible-playbook -l local deploy/setup.yml

Configure

There are now two files on the machine that need to be edited, /etc/exventure.config.exs and /etc/nginx/sites-enabled/exventure. The nginx config file just needs to know what it's domain name is. The exventure.config.exs file needs more modification. Any empty string "" should be filled in with the required information.

You should set up SMTP so your game can send emails, this will also require setting the from address.

The endpoint and networking lines both need to know the new domain name for your game as well. You should also connect your game to Gossip. It's more fun that way.

Certbot

Once you are configured, restart nginx and then configure SSL via certbot.

sudo systemctl restart nginx
sudo certbot --nginx

Follow the prompts for configuring your new domain. You should set the game to redirect any HTTP traffic to the HTTPS port.

Deploying

Next you can deploy your game, the first deploy you should seed as well. ExVenture comes with a simple deploy script that will handle most of this for you. The last argument should be the fully qualified domain name that you previously set up. It will try SSH'ing in as deploy.

Before deploying, make sure to generate a production release with the release.sh script. Note that this needs to be run on a linux system. You cannot deploy an erlang release built on a mac to a linux remote.

./release.sh
./deploy.sh --seed my-game.example.com

On future deploys, you only need to use

./deploy.sh my-game.example.com

A Remote Server

The previous steps got a local VM ready. Getting a remote machine is not that much harder. I recommend using Digital Ocean as your VPS of choice, ExVenture World is hosted there.

Once you create a droplet (using Ubuntu 18.04), SSH in and install python.

sudo apt install python

Next setup a new host file named deploy/host_remote with the following:

[remote]
the.ip.or.domain
ansible-playbook -i deploy/host_remote deploy/setup.yml

And the rest is the same. Configure the game, configure certbot, generate a release, and deploy it.

Conclusion

Hopefully this helps you get a remote ExVenture game set up and running. If you have any questions please come by the Discord group.

Creative Commons License
This site's content is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise specified. Code on this site is licensed under the MIT License unless otherwise specified.