Maz Development Directory

Ignorance is a Blister

2023-11-28T20:50:23+00:00

Introduction

This is yet another reboot of the attempt to dish out more fast-paced, short-form content in order to keep the ball rolling. Previous attempts have not been successful, so my hopes are not high, but I’m going to give it another shot anyway.

That’s it for the introduction.

Good Ignorance

The saying goes, “Ignorance is bliss”, and for good reason. Ignorance of bad information is generally good. If you are not aware of certain suffering that is going on in the world, or some ugly truth about a person you know, you are in bliss.

This also applies to programming to a certain degree in certain contexts. For example, PHP. I’m really sorry, but if you were a developer back in 2009, and you started your programming journey with PHP, you probably had to unlearn a lot of stuff to get back on track, in order to then improve from there. It was not a language that promoted clean and simple code design.

Being ignorant of PHP in the 2000s was a bliss.

The final good ignorance I’ll mention in programming is the ignorance of future problems. Back when I was an inexperienced developer, not even aware of Git, I used to ship “apps” on regular basis. They were simple apps, however, they were useful in their own right. They provided a good starting point to grow from. I feel like I was shipping for the sake of shipping, even though the final products were imperfect (to say the least).

Nowadays, I spend a week figuring out which VS Code extension will work best for Python linting before I even write a single line of code… You may not see it as an “ignorance” problem per se, but I beg to differ. The fact that I know about linting in the first place and care about it is partially the reason why I’m having such issues.

Bad Ignorance

The main reason for this post, however, is to talk about the bad ignorance that has also plagued me recently in multiple ways.

SQLAlchemy

I made a career transition from an iOS developer to a Python/Fullstack developer when I started my current job. Needless to say, I had a lot to learn, and little time to do so. Granted, I already built a backend for my Dama King game, however, it was in a hustler fashion, just to get the game out the door. I didn’t spend the time to learn each technology I used in depth.

My job is in AI/ML, so naturally, I had to use python for most of my work. Thankfully, I already built many internal tools in Python as an iOS developer, so I was familiar with the language. However, I never really had to deal with a database in any significant way. So, the first problem I was faced with was, which ORM should I use?

Hold on, which ORM to use? What about using raw SQL? That wasn’t even an option I entertained at all. I thought it was a no-brainer to use an ORM, period. Now, given that an ORM was a must, I had been to many python meetups, and everyone swears by SQLAlchemy. So, I went with that. I remember posting a message in a group chat asking for advice on how to learn the basics of SQLAlchemy for CRUD, and one guy said, “O V E R K I L L L”. I was adamant to use it anyway.

Here is where the lesson begins. I didn’t spend nearly enough time learning SQLAlchemy, and jumped in head first, flat on my face.

My biggest mistake was not learning SQL first. “Seriously, you don’t know SQL?”, I hear you say. I knew the basics, just like any other developer, however, the nuances of SQL were lost on me. Window functions, CTEs, JSON, useful EXISTS subqueries, etc.

Why was that an issue? Well, I kept fighting SQLAlchemy, trying to make it do things that simply don’t make sense in SQL, so how can SQLAlchemy do it? Two, or three, years into my job is when I finally took a step back and decided that I should craft SQL queries at a lower level to control exactly what would SQLAlchemy would generate. Thankfully, we were using SQLAlchemy 1.14, which started a transition to SQLAlchemy 2.0 syntax. This was great since SQLA 2.0 uses a more SQL-like syntax, for example:

import sqlalchemy as sa

# we have a repo class that wraps sqla engine.
with repo.session() as session:
  q = sa.select(MyTable).where(MyTable.id == 1)
  session.execute(q).all()

War Stories

My biggest problem was with SQLAlchemy. The mere fact that it lingered for years before I finally realized I should learn me some SQL, is just mind boggling. I have similar war stories, albeit not as bad, with other technologies.

For example, I don’t know why I felt I should try hard to always use the docker images provided by the official maintainers of the technology I’m using. I used FastAPI’s, jupyterlab’s, and other images provided by the maintainers, which I would then extend to add my own customizations. This left me wasting a ton of time trying to figure out how each base image worked, and how to adapt it to my needs. Soon enough, we built our own solid base image, which became the foundation for all our services.

Another example is Yup (a schema validation library for JavaScript). I thought it was simple enough to use without reading the docs, nor preparing a sandbox environment to test it out. Boy, was I wrong. I wasted hours (days?) wrangling with it, and each time, I tested it against the UI with Formik, which made it even more difficult to debug. I eventually created a sandbox environment and used that to test out my schemas before using them in the project. The speed of feedback was critical in this case.

Conclusion

In conclusion, the lesson I impart to you is to realize when you are wasting more time than necessary on a technology that you’re treating as some sort of black box, when you should instead take a step back and learn the basics first.

Skinning a Cat

2023-11-25T20:50:23+00:00

Introduction

I find the saying “There’s more than one way to skin a cat”, just as much as the next guy, quite morbid and gruesome. And yet, I like to use it and tell myself that people aren’t really skinning cats for sport or professionally, it’s just a saying. So the cat we are going to skin today is Visual Studio Code.

I can’t believe I somehow, completely off-handedly and irresponsibly, added yet another opinion to the heated question of whether VS Code is an Editor or an IDE. I’m making a case it’s a cat, apparently.

Digging Tunnels

We recently got hold of the hottest piece of hardware on the market these days.. The glorious A100 GPU. Granted, the H100 eclipsed it, and next year we will see the H200 and B100, however, for simple people like myself, the A100 is the best hardware I could realistically expect to work with. Now, we have it.

This beast of a hardware has been installed in a data center at an undisclosed location somewhere on earth, and attached to a Linux VM for us to use it through. As soon as we were given access, we immediately fired up our browsers, navigated to the JupyterLab service we previously prepared on this VM, and started mining crypto in the name of “Benchmarking the System”.

JK, of course, I would never admit to such illegal use of company hardware pfft.

OK, in all honesty, we simply logged in to JupyterLab, pulled some 7B, 13B, 30B, and 70B parameter LLMs, and let ‘em rip on the GPU.

We .. Were .. Blown .. Away.

The fact that high quality models, like the llama2 13B, loaded pretty quickly, and started giving us wealth of knowledge to use was incredible. My brain felt like fireworks were going off in it as I was thinking of all the possibilities. At the same time, I also realized that JupyterLab isn’t necessarily the ideal environment for maximum productivity. Users shared the same environment, it didn’t have powerful extensions like VS Code, and it was restricted in many ways.

It was time to dig some tunnels.

The Cat

Visual Studio Code was our cat of choice. We primarily use it for Python, but it also works well for Javascript development when we need to work on the frontend. I started exploring remote development on VS Code back in 2021, I think, and at that time, my goal was simply to write code on the go on my iPad, saving me the hassle of carrying my laptop around (since I was already carrying my iPad around).

Code Server

I started with Code Server, which is a project that aims to bring VS Code to the browser. It was a bit annoying to setup (running it on Docker, deploying it, connecting my stuff, security concerns, etc), but it worked well enough. My biggest pain point by far, though, was the lack of extensions. I had started using Github Copilot to write most of my code, as well as other Microsoft extensions for Python development, and none of them were available on Code Server.

Another serious issue I had was syncing my projects across devices. I always had to save my changes, push to git, pull from the other device, even though it was just me working on the project. Soon enough, I was able to get access to VS Code private beta of their remote development extension, aptly named “VSCode Server”.

VSCode Server

VSCode Server was a retaliatory move by Microsoft to counter the success of Code Server (I speculate). It unlocked the full power of VS Code along with all the extensions we know and love. It was also much easier to setup, since it was just a VS Code extension. And yet, there was still a catch.

You see, when I switched to VSCode Server, I needed a safe way to connect from my iPad to whichever machine I was working on. Luckily, I stumbled upon NordVPN’s new feature at the time, Meshnet. It was an additional feature that made any device connected to NordVPN appear on the same network, as if they were all connected to the same router. :mind-blown:

If only it worked as reliably as it sounded.

Sadly, after the initial euphoria, I realized that the connection was unreliable, and I would often lose connection to the remote machine. After some time, I gave up on it, and came to terms with the fact that I would have to carry my laptop around with me.

Same Cat, Different Tunnel

Sorry for the unannounced segue regarding my previous attempts at remote development outside work, but I felt it was necessary to set the stage for the next part of this post.

So, let’s get back to the A100 GPU and our local development needs.

Corn Kernels

Since we were already running JupyterLab on the VM, the immediate solution that came to mind was to use JupyterLab’s VS Code extension to select a kernel running on the VM, and use that to run our code. This worked seamlessly, and honestly, I thought I as done then and there.

However, we quickly realized that code referencing broke. VS Code showed squiggly lines under all the imports, and we couldn’t navigate to the definitions of any of the functions we were using. This was a deal breaker for us.

Another side effect was the lack of structure parity between the local project and the actual environment. For example, if you want to import a local file, or use an LLM, you need to reference it from the remote server, which made it very awkward to work with.

Being a Good Host

At that point, I was like, no problem! We will just install Python 3 on the VM, and use that as an SSH host similar to how we were using WSL2 on Windows locally.

Since we are using RHEL 7, it had python 2.7 installed by default. So, I fired up perplexity, and it gave me a perfect, step-by-step guide on how to install Python 3 from source. I had to do it this way, since yum was outdated, and I didn’t want to risk downloading potentially incompatible packages from other sources. The installation went fine, until I started the REPL, and it turned out I now needed to install/compile openssl, sqlite, and a bunch of other stuff. (╯°□°）╯︵ ┻━┻

The Devil’s in the Container

Since we deploy all our services on Docker, it made sense to give Docker a shot at this. I wanted to try using Dev Containers for a while, but never had a good reason to do so. This was the perfect opportunity.

I struggled for a bit at first, since the default dev container settings I had was showing a warning as deprecated, while the new format didn’t yet have enough documentation and examples online. I eventually figured it out, but it was just for running a local container as a dev environment. I needed it on the GPU VM.

I first tested the obvious way, which is to connect to the host through VS Code SSH, then manage the dev containers. This kinda worked, but now you are in this inception-like situation where you are connecting to a VM through SSH, to run a container also on the remote host. Not a big fan.

I then researched on how to use the DOCKER_HOST environment variable to connect the local docker client to a remote docker daemon. This had the potential to eliminate the need for the SSH step, however, it introduced a new more serious problem. When VS Code tries to prepare the dev container, it symlinks the local project to a folder, which is then mounted to the container as a volume. We can’t symlink the local project to the VM, obviously.

The Final Boss

As the weekend came around, I was thinking about what to try next, since it seemed like such a simple problem! As I was thinking about the simple facts and the problem I’m trying to solve here, I realized that I could just have a remote container which the local VS Code can SSH into, just like WSL. I mean, just because we will be running the code in a container, doesn’t mean we have to use dev containers!

This is the step we are at right now, honestly. Just setting up a simple container that exposes an SSH server, and allows VS Code to connect to it directly, without the need to go through the host first. An initial PoC showed some promising results.

Conclusion

I’ve been reading Elon Musk’s biography, by Walter Isaacson, and I have to say, it’s been a very inspiring read. I shall impart a prat of the book I found particularly useful to apply at work:

The Algorithm:

Question Every Requirement
Delete as Much as Possible
Simplify and Optimize
Automate
Accelerate

Appease Your AI

2023-08-29T17:55:23+00:00

Introduction

Who isn’t talking about AI these days? This topic has made its way into every conversation, in every field, in every way imaginable (and unimaginable).

The obvious areas where AI is mentioned could be automation, robotics, and perhaps the hardware accelerator manufacturers. These areas are basically intertwined with AI, no doubt about that.

Then comes the second wave of areas, where AI applications made sense, not necessarily the underlying details. This ranges widely, from Software Engineering, health care and wellness, education, all the way to weaponizing AI at the military, scammers / bots, and other nefarious applications.

Nowadays, you can bring up AI in every conversation imaginable. Building a toy for kids? Stick an AI in that and have it talk to your kids so you don’t have to. How about making sandwiches? You got to pre-scan those ingredients with AI. Also, have it optimize your menu over time to drive sales.

Just to be clear, I am not saying AI is new, or it hasn’t touched our lives in profound ways yet. It has, for decades now, even if most people are oblivious to this fact. However, the surfacing of this topic and more ubiquitous applications is what has changed recently.

So, Let’s Talk AI

Writing code alongside Github Copilot has influenced the way I write code. I can’t tell for sure whether it’s good or bad in many cases, but it’s a bit scary as probably millions of programmers out there are experiencing this affect as well, as we all unknowingly converge because of the advent of code generating LLMs.

Bad Copilot

When using Github Copilot for any reasonable amount of time, you will quickly face an undeniable shortfall. It can only suggest code where the cursor is pointing. I know, I know, you can use Github Copilot Chat / ChatGPT / … etc to sort of work around this, but that really breaks your flow. I like writing code in small chunks, and would expect Copilot to mimic the way I write code, namely, but moving the cursor and making changes in several places to achieve the desired result.

Instead, we get this:

import os
from typing import Any

def run_some_code(config: dict[str, Any]):
  some_value = os.environ.get("FOO_BAR")
  output_file = "file.txt"

  # check if the file exists
  # if os.path.exists(...)

Copilot will invariably use the available imports to deduce that you probably want to use them as oppose to other, potentially better, approaches. In this case, I usually stick to using pathlib.Path. Honestly, I’ve avoided pathlib perhaps once in my career due to performance concerns, but in all other cases, the ergonomics beats os.path 10-0.

Another issue that may arise is the lack of proper type hints. Since we only import Any from typing module, Copilot will use Any and built-ins exclusively to annotate your code (99% of the time). Many cases where I wanted it to use Callable it would just use callable built-in instead, for example.

Good Pilot

I actually started to change the way I write code in order to make copilot more capable and expressive, so to speak. Here is one obvious improvement over the above:

import dataclasses as dc
import datetime as dt
import typing as t

@dc.dataclass
class Foo:
  date: dt.datetime
  callback: t.Callable[[], t.Any]

By importing the modules themselves (and using aliases to reduce verbosity), we have effectively prepared the environment for Copilot to be able to work more efficiently, making me the copilot, effectively!

This is a very simple change that anyone can adopt to their code style, and would pay back dividends in time saved trying to steer Copilot to use certain imports.

First of Many

This is just one very obvious impact using Copilot has had over my code style, while I am sure many less obvious have already made their way in. Personally, I am embracing it as a good thing. At the end of the day, code is not really art. It is instructions arranged in specific ways to serve a higher purpose, not more.

I keep trying to think of other possibilities to better leverage Github Copilot. A perhaps extreme example would be to force myself to document each function with a Copilot-friendly docstring that should produce the desired result + unit test without any intervention from me besides iterating on the docstring.

Conclusion

Whenever you feel useless in the face of AI’s quasi-infinite potential… Whenever you see yourself unable to cope with AI’s lighting-fast advancements… Whenever you want to just shut all this out and live in simpler times…

Remember…

It’s already here, and there is no stopping it.

Verdict: It's a Match!

2023-08-20T16:55:23+00:00

Introduction

Second blog post since forever, and I have no idea what this one should be about. The main problem is I”m writing this a I finish my workout, so I don’t have much going on in terms of setup, but that’s fine.

After the rant in the last post, I would like to aim for a more informative post this time. Something with more “meat”, so to speak. It’s not much, but perhaps I’ll share my experience using Python’s match statement; introduced in Python 3.10.

The Primal Age

In a not so distant past, writing code for checking the structure and types of objects was a huge pain in Python. Here is a real-world example of an object schema:

{
  "job_type": "training",
  "train_config": "some config",
  "min_examples": 10,
}

Intuitively, job_type value can be one of several predefined values, and based on the job_type, we would like to invoke a custom flow, or banch of code if you will. How would one normally handle this in Python?

def handle_job(job: dict):
  job_type = job.pop("job_type")
  if job_type == "training":
    train_job(**job)
  elif ...:
    ...

Looks fine to me, what do you think?

The main concern that screams danger at me is the full trust we have that the job is indeed a dict and that is has the key job_type. Granted, fixing the potential KeyError is easy, just add a default and you’re golden. Regarding the type assurance, you have to have some level of trust, especially if this is an internal function that you fully own.

This code is actually perfectly fine, and I wouldn’t necessary bark at it during a code review.

Let’s take another, rather similar example:

# snippet from a function that takes an SQLAlchemy orm type and a "dynamic filter",
# and returns a SQLAlchemy filter expression.
#
# e.g.
# ("date", ("2022-01-01", "2023-01-01")) -> orm_type.date.between(date1, date2)
#
if isinstance(filter_, (list, tuple)):
  if len(filter_) == 2 and isinstance(filter_[0], str) and isinstance(filter_[1], (list, tuple)):
    return getattr(orm_type, key).between(filter_[1][0], filter_[1][1])
  elif ...:
    ...
else:
  return filter_

If you want to go ahead and gouge your eyeballs out, please, be my guest. I started this project when my dependencies required me to use Python 3.8, and then soon after Python 3.9. I remember how badly I wanted to “matchify” this code…

I mean, even if one is more inclined to make it self-documenting, it may look something like this:

if isinstance(filter_, (list, tuple)):
  key = filter_[0]
  is_between_expr = len(filter_) == 2 and isinstance(filter[1], (tuple, list)
  if isbetween_expr:
    ...
else:
  return filter_

Yikes, doesn’t look any better to me, honestly.

Eye Candy

match filter_:
    case (str() as key, (start, end)):
        return getattr(orm_type, key).between(start, end)
    case (str() as key, str() as value) if re.match(r"^%[^%]+%$", value):
        return getattr(orm_type, key).like(value)
    case (str() as key, value):
        return getattr(orm_type, key) == value
    case (str() as key, str() as operation, _ as value):
        op = getattr(operator, operation)
        return op(getattr(orm_type, key), value)
    case _:
        return filter_

Huge sigh of relief! Not only is the code succinct, safe, expressive it’s also self-documenting. Keep in mind this code is deep within the code base, after Pydantic has done its validation on the input, so no need for us to be pedantic about it (pun intended). Yet we really do need to check types and structure to properly determine what to execute.

I can hear you already from shouting from behind the screen:

“Actually, the problem here is the spec itself. It’s too vague.”

Fair enough. If we had passed the actual operation as some sort of OP_CODE type of thing, parsing would definitely be less hairy. To be completely honest, this code is not used anymore, lol. But for that time when I needed to quickly have a simple spec to manage sql filters dynamically, match had my back.

Match Madness

For the final piece of this post, I present you with an actually useful use case for the match statement:

# Recursively parse an SQLAlchemy ORM type, mapping everything to primitive types.
match obj:
    case dict():
        return {
            _to_dict_or_list(k, exclude_refs): _to_dict_or_list(v, exclude_refs)
            for k, v in obj.items()
            if not isinstance(k, str) or not k.startswith("_")
        }
    case tuple() | sa.engine.Row() if hasattr(obj, "_fields"):
        return {
            _to_dict_or_list(k, exclude_refs): _to_dict_or_list(v, exclude_refs)
            for k, v in zip(obj._fields, obj)
        }  # noqa
    case list() | tuple():
        return list(_to_dict_or_list(v, exclude_refs) for v in obj)
    case Enum():
        return obj.value
    case _:
        return copy.deepcopy(obj)

Honestly, this looks like a monster. The value of this code will also greatly diminish once we migrate to SQLAlchemy 2.0 with Mapped attributes, but the main use-case still holds true.

The idea is that working with arbitrary data structures, with various shapes and sizes, can get out of hand very quickly. Being able to break down the conditions that need to be met into readable statements goes a long way. This code could use a bit of cleaning as well, perhaps using local functions to keep each case body as simple as possible, so there is still room to improve beyond just using match.

Conclusion

I am all for Python’s steady march towards the functional paradigm. The less side-effects, mutation, inheritance, and other monsters you can avoid, the better off we’ll all be. I just wish pipes come very soon. They just can’t come soon enough, that’s for sure.

Playing the Part

2023-08-19T12:55:23+00:00

Introduction

Boy, has it been ages since I last wrote a post. I can hardly recall my style, even, however that’s not really an issue as we’ll see.

Out With the Old

As you may (or may not) have noticed, the blog is finally migrated to a sensible static site generator, namely Pelican. I’ve been using Jekyll for over a decade! Albeit a decade of frustration, unfortunately.

Look, I am not one to complain, especially when it comes to open source software. As an open source contributor myself, I know how hard it is to maintain a project, let alone a popular one. However, at the end of the day, you have to be entitled to your own opinion, and sharing that opinion will help others navigate the open source landscape better.

So, what’s wrong with Jekyll?

The Downsides

First and foremost, Ruby.

I’ve played around with Ruby in the past, and my experience was generally pretty pleasant. I’m a big fan of Elixir, for that matter, and its syntax is heavily inspired by Ruby. Heck, half of the Elixir core developers are recovering Rubyists. Not only that, but I’ve also used Ruby with respect to iOS development with projects like Cocoapods and Fastlane.

Alas, I’ve never really truly understood Ruby tooling. There is the rake thing, then there is the Gem thing, and then there is the Bundler thing. Every single time I came back to my blog in a new environment, I had to struggle with these tools. I mean, don’t get me wrong, I don’t mind playing with a new language for fun. I’ve written small projects with Go and Rust, but the tooling they provide is phenomenal.

Next up, Jekyll itself (obviously).

As far as I can tell, ever since Github adopted (acquired?) Jekyll, development of the framework slowed down tremendously. For a static site generator, keeping with the times is crucial, to say the least. Even if the framework wants to be minimalistic, constant refactoring and simplification for reduced complexity and easier onboarding is paramount. I mean, Jekyll was so unapproachable to many in the past, to the point that it lead to the development of Octopress. (Granted, Jekyll eventually got a bit better over time and the need for Octopress greatly diminished after that).

Other issues I had with Jekyll were mostly inconveniences rather than deal-breakers, however, you know how a thousand paper cuts can kill you. These range from the performance of Jekyll itself, to the insane number of dependencies, some of which are just a pain to fetch/build (I’m looking at you, nokogiri), and the lack of in-depth documentation. (Just to make sure, I opened the Jekyll website to see if it was improved, and sure enough, it’s the same website from a decade ago!)

In With the New

Enter Hugo!

Yes, you heard that right, I migrated to Hugo. The dazzling hotness of Go, in combination with the clean website/docs of the framework, made me jump in head over heels. And let me tell you, it was super exciting! At first anyway.

Migrating to Hugo was generally a pleasant experience. They had importers that support importing a site from Jekyll, so that took a huge burden off my plate. Next, I was just astonished by the performance. The whole site was being fully generated in mere milliseconds every time I made a change. I was in love… Until.

Sadly, the honeymoon phase was over within mere days. I don’t even recall exactly what went wrong, to be honest. It was just too hard for me to work with, constantly looking up documentation, spending time trying to figure out what are the supported functions in the template engine, and so on… For example, in order to format the date object, I was searching for the date specifiers for Go, and to my astonishment, they use an actual date for formatting.

This gave me an immediate lightbulb moment. What if I used a static site generator built with a language that I know and love?

In With the Newer

Enter Pelican!

I have to say, when I was first exploring Pelican, I was extremely hesitant to throw away the efforts I spent on the Hugo blog migration. Pelican uses reStructured text as the default format, which rubbed me the wrong way as a first impression…

Start tangent

Speaking of rubbed the wrong way, it is actually the exact same feeling when I first learned about pyproject.toml. I’ve been using Python for years, and all of a sudden, there is this new markup language I’ve never used before, with a rather silly name, too, and I’m suppose to adopt it to manage my dependencies and project? The inner resistance was very high at first, but as is always the case, I had trust in the seasoned developers of the PSF, and decided to lower my guard and approach it with an open-mind. Few years later, this is perhaps the cleanest, flat-structure markup language I’ve worked with.

End tangent

Pelican also didn’t use frontmatter, like Jekyll and Hugo. Finally, I was having trouble with the bare-bones, minimalistic core that Pelican aims to be. It felt like I’ll be spending most of my time hunting for plugins to make my workflow work. Thankfully, I was wrong.

Great Artists

After getting past the initial learning curve of Pelican, which mostly consisted of figuring out the structure and settings (like what are themes, how are static files added, what is the best configuration for development vs. publishing, …etc), I realized that I could just search Github for other Pelican-ians (?) to see what their set up looked like and if I can, ehm, “borrow” some of their knowledge. Surely enough, found the indispensible pymdown-extensions. Basically, pymdown-extensions is to Python markdown project, as oh-my-zsh is to the Zsh project.

Armed with these fantastic extensions, the migration progress pace started to pick up. I spent way more time than I should polishing the code blocks, and trying out different themes and styles for that matter. Code blocks have been an obsession that I haven’t been able to crack before. I’ve tried to embed Github gists, different highlight frameworks, both in Javascript and pre-rendered, but was never really satisfied by the outcome. That is, until now.

Closing the Loop

Remember that remark in the introduction? About how not remembering my style is not really an issue? Well that’s because of the following…

At the beginning of the migration, I was going to import all the Sass and HTML templates setup from the old site. I wanted to finish this migration as fast as possible, but quickly realized a few main points:

There was A LOT of unused crap laying around that I never really bothered to clean up in the past. Most of it was because of using a Jekyll theme as a base, and then heavily modified it to the point that I was probably using only 10% of the original theme.
I may not look like it, but I am a web developer now. Indeed, in the past 3 years, I’ve engrossed myself in web development and I’ve become much more familiar with aspects of web development than before. I could easily see the many issues that existed, and how to quickly fix them.
Many of the assumptions I had around the structure of the blog and layout were simply wrong, and I was gonna port all that to this new, fresh beginning without a second thought. Alternatively, I could take it piece by piece and evaluate what stays and what goes.

Hence, I shall not simply keep carrying the baggage of the past for the mere legacy of it. It shall be re-evaluated and, if needed, improved upon … for a better future(?).

Conclusion

I’m wondering, as I always do, is this post a one-off, or is it a proper reboot of this forgotten blog? It pains me to see the missing years in the home index, as well as the existence of just a handful of posts in some years… But this pain is but a motivator … for a better future. (Maybe I should try openai whisper to blog with just voice. That will be interesting.)

Async Pipes

2022-03-10T03:55:23+00:00

Introduction

Over the past couple of posts, I explored basic parallelism in Python with the help of Ray. Since then, I’ve adopted the library in my work and built some pretty cool things with it, all of which I would like to cover in detail someday:

Actor Pools: I’ve touched on the topic briefly in the last post.
Worker Queues: Implementing a robust queue with progress updates around Ray workers.
Async Pipeline: Implementing a flexible, reusable data processing pipeline with Ray.

Since I would like to cover the Async Pipelines in this post, let’s see what they’re all about, shall we?

The Shrinking Middle Class

Before we can talk about async pipelines, we must first talk about the shrinking Middle Class.

I follow U.S. politics simply because it’s interesting to me. You could say it’s the same reason I watch shows on Netflix, or browse YouTube for Mario speedruns. One key topic that keeps being brought in U.S. politics up is the problem of “The Shrinking Middle Class”.

Given the existence of a Middle Class, this means that there is also an Upper Class and a Lower Class. The Upper Class are your CEOs, Investors, and Leaders of the nation. Although they are a small percentage, but effectively they are responsible for most of the jobs in the country.

The Lower Class, on the other hand, are your student-debt ridden youth and immigrants who are left to pick up the slack, doing meaningless jobs that no one else wants to do.

So, what is the problem if the Middle Class is shrinking?

Well, people in the Lower Class might lose hope of one day reaching the Middle Class, given its inevitable demise. The people of the Upper Class won’t have people doing the interesting work for them, slowing down innovation and economical growth.

I know what you’re thinking right about now. How on earth is this remotely relevant to the topic of this post?

Glad you asked.

Scaling the Middle Class

Aside from my need for parallelism in Python to simply scale a worker pool that handles API requests, I also needed that same pool to process work within a data pipeline.

The data pipelines I have usually involves one (or a few) data generators that are simply Python generators that generate data for processing. (Say, generate, one more time…)

That’s your Upper Class.

Data is then consumed by workers that perform some interesting processing, generating a result. This processing is where the bulk of the CPU cycles go to.

That’s your Middle Class.

Finally, a collector collects the results and commits them to a database, taking care of any data inconsistencies and reporting the results at the end.

That’s your Lower Class.

And hence, our goal today is to scale the Middle Class only, in order to process as much data as possible, maximizing CPU utilization (productivity) along the way.

Fan-out & Fan-in

Let’s try our hands on a basic approach to tackle the problem statement above, shall we?

import time
from concurrent.futures import ThreadPoolExecutor

def work(data):
    time.sleep(1)
    return "done"

def callback(future):
    print(future.result())

with ThreadPoolExecutor() as executor:
    for data in data_generator:
        future = executor.submit(work, data)
        future.add_done_callback(callback)

This is fine … except, the whole place is on fire, and I’m too busy supressing the cringe to care. Let’s ignore the fact that we are using threads here, because that part is just for demonstration purposes.

My issue with this code is the nesting and lack of clarity. I also believe the data_generator will fill up the executor as fast as it possibly can, which can lead to memory growth. Back-pressure to limit the data generator output would be nice, too. This problem also makes it harder to track our progress of how much data was processed.

Finally, we are in an era of async / await. I don’t even want to try throwing that into the mix, in fear of reaching depths for which there is no escape.

(DISCLAIMER: I have to admit though, I didn’t expect to come up with a palatable solution on the fly for this problem, lol. I spent way too much time solving it in our project)

I Hereby Declare

… the scalability of the Middle Class!

await async_flow.pipeline(
    async_flow.Layer(data_generator),
    async_flow.Layer(work, 8),
    async_flow.Layer(callback),
)

async_flow is the basic utility I prepared in order to reuse the scalability semantics across the project. The details of the utility is too much to cover in this post, but it basically has two types:

Layer: Declaring a function, optionally scaling it as needed.
pipeline: To declare your Layers in for execution

Not only is it a pleasure to just look at, but it also supports back-pressure, because the Layer objects passed in are stitched togething using asyncio.Queue. This way, if the work Queue object is full, for example, the data_generator will patiently block on the await queue.put(data) statement till it’s free.

Conclusion

Although I meant to cover a little more details about async_flow, or at least give a full working example for that matter, I spend too much time on the introduction and now it’s getting late. Regardless, the internals are worthy of a whole post on it’s own.

Balance

2022-01-19T20:30:23+00:00

Introduction

In the previous post, I wrote about parallelism, and towards the end, mentioned Ray, the ultimate python package for parallelism.

I played around with it today, and wanted to give my take as well as describe the new challenge that I am currently stuck on.

Prelude

Ray is just so much fun. I am so glad I found it.

Want to define a function that can execute in the background?

@ray.remote
def add(x, y):
    return x + y

await add.remote(1, 2)

Just how sick is this? Seriously? That function could also execute in a distributed manner, on some other machine entirely if you wanted to.

How about Actors? That must be more challenging to write, right? … WRONG.

@ray.remote
class Actor:
    def __init__(self):
        self.state = 0
    def increment(self, x):
        self.state += x
        return self.state

actor = Actor.remote()
await actor.increment.remote(1)
print(await actor.increment.remote(1))
# Output: 2

Sateful actors, with a few simple lines of code. Just love it.

The Catch

So far, so good. Where is the catch?

I haven’t been able to find a built-in approach to dispatch work to a pool of workers and await each result separately.

I mean, there is ray.util.ActorPool, however, its API is very limited. As far as I can tell, the main APIs are ActorPool.map and ActorPool.submit. ActorPool.map makes sense if you have a bunch of data to process, but ActorPool.submit is more for one-off tasks.

Given the above, ActorPool.map definitely won’t do for my usecase which involves submitting work that is coming from an API request. ActorPool.submit sounded interesting, however, I have no idea why they chose to make it not return anything. That means, once you submit a task, you have to go search somewhere else for the result :ugh:.

The Solution

I will now try to conjure up a solution as I go.

At first, I noticed that ActorPool had one more cool API up its sleeve. ActorPool.pop_idle. That would work for me, if I can just try to pop an idle actor, do my work on it, and return the result. Unfortunately, if there isn’t an idle actor at the time of the call, it will return None. I wanted a blocking solution instead.

After reading the source for ActorPool, I realized the usecase I want to use the actors for is completely different. For example, I would like to keep the async api.

… After a bit of pondering, I realized I could just use a blocking queue to achieve the desired result. Pop an available actor, use it, then put it back. New API requests that come in will block until an actor is available in the queue.

Of course, I will prepare my own abstraction around the queue of actors to provide a simple API similar to ActorPool, ProcessPoolExecutor, etc.

Conclusion

How far will this streak go?

Not Fast Enough

2022-01-18T20:30:23+00:00

Introduction

Why am I writing after such a long absence? Wrong question. Why haven’t I been writing for so long?!

I could just sit here and give all kinds of excuses and reasons as to why that is, but the truth is … I slipped. I think.

Programming was pretty much my life back in the day. I was writing code and learning on such a constant pace, it only made sense to keep writing these blog posts as a way to channel the residual energy. Not so much now.

Not the time to cover this, it’s time to get my time’s worth in reviving this zombie of a blog!!

The Mourning

As previously expressed, emphasized, and then over-emphasized, Elixir is the dream language for me. I wouldn’t waste time nor effort on other programming languages if I had the choice, alas, life ain’t that rosy.

In my current project, I have to settle with Python, which would be my second best choice… Although that might change given the introduction of the Actor pattern in Swift. Back to the point.

Python is a very productive language, generally speaking. There is not much worry about in terms of syntax, however, other quite critical things are missing. Paralellism.

The Multiverse

In python, you could try and make your code “parallel” by using threads or asyncio, but you shall not achieve much success. Threads are haunted by the all-evil GIL, which is a global interpreter lock. It won’t actually allow mutiple threads to run at the same time, and it will also block the main thread from doing anything else.

As for asyncio, it is single-threaded by definition, so it is not a parallelism tool. It is, however, useful for concurrency. That leaves us with the only remaining viable option, that is multiprocessing.

The Future

Hello, this is Maz from the future! Maz from 15 minutes ago wrote the post up till now, and he already made a mistake!

Multiprocessing is indeed an option, but not the only remaining option. You could do distributed computing, just as an example. Actually, after I wrote the last line about multiprocessing above, I thought to myself how the multiprocessing module is extremely limited!

You can’t really send local funtions/classes, as the pickling process to send the information to the subprocesses uses pickle, which only sends a reference. Dill, however, is a third-party module that allows you to send local objects. Instead of using dill on top of multiprocessing, however, might as well find a all-in-one solution that replaces multiprocessing usage altogether.

Just to mention another annoying limitation of the multiprocessing module, it doesn’t allow spawining nested multiprocessing pools. This is because the multiprocessing pool creates daemon processes, which aren’t allowed to have children.

Follow the Light (Ray)

Ray is a python package that promises to keep things simple, while still being able to do some pretty cool things. Right off the bat, they seem to have the actor pattern as a feature. Besides that, they allow you to structure your code declaratively in order to make it executable in a parallel environment. After that, it’s totally up to you how you want to parallelize the execution. You could use threads, processes, or even distributed systems.

Gotta hand it to them, I am sold.

Conclusion

I had to clear my head, lay out everything on the table, hence I decided to write again. This was extremely helpful in articulating my thoughts and reaching a conclusion on what I should do about this parallelism problem. Perhaps I should do this more often.

I leave you with a nice quote:

“The best way to predict the future is to invent it.”

Umm, no … The above quote was actually predicted by github copilot. I wanted to write a different one:

“If this were easy, how would it look like?”

– Tim Ferriss

A Quick Abstraction

2018-11-02T22:30:23+00:00

Introduction

As software engineers, our job is to make sure computers behave in very specific ways when the software we have built is being executed. Now, this sounds extremely trivial, but this is what it really boils down to…

Unfortunately, the tools we are provided to make this happen are extremely flawed and rudimentary. Even if I’m passionate about software and programming, building software shouldn’t be so hard. But it is, and we end up making tens, if not hundreds, of micro-decisions when writing software..

Perhaps the noblest of all goals a software engineer could ever cherish is fighting complexity. Less complexity leads to so many wonderful things, some of which are less chances for error, easier maintenance, and many times, less code overall.

An important weapon against complexity is abstraction. If done right, a proper abstraction could help you break down a very complicated problem into it’s essential parts, and make it much more manageable when looking at it as a whole. I wanted to demonstrate one example of that today.

Perform Once

Here is an exercise for you, dear reader. Say we have some class, and within that class, we have some code we want to execute only once during the whole life of its object. How would we achieve that?

… Think about it for a bit.

Well, the answer here should be relatively easy and simple. Put that code in the constructor of that object. Done. The constructor is guaranteed (by the compiler) to be executed only once. Not bad, so, let’s throw in an extra piece of the requirement.

Now, what if that code we want to execute only once, depends on some condition. This condition might be true when the object is initialized, meaning we can execute the code in the constructor as we did previously, but it may also be the case where the condition is initially false, and at a later time becomes true?

… The immediate answer should be clear.

Generally, the inclination would be to check for that condition as it changes, cache it somewhere to detect how it is changing, and in order to make sure that piece of code is executed only once. Something like:

func conditionChanged() {
    if codeDidNotExecuteBefore {
        executeCode()
    }
}

The code above is really bad, and is the worst possible solution for this problem, probably. This doesn’t protect the function from other possible call sites, and decouples the condition from it’s actual purpose. Let’s fix that:

func executeCodeOnce() {
    guard codeDidNotExecuteBefore else { return }
    codeDidNotExecuteBefore = true

    // ...
}

Much better. There is no way we might accidentally run the code twice this way (assuming single threaded environment). Also, the condition is tied to the code it’s guarding.

Now, we can do even better, by getting rid of this explicit branching. Branching is generally bad and if we can avoid it, it’s much better, as it’s simpler to make sense of flat code that executes all sequential statements every time. Take a look:

func conditionChanged() {
    executeCode?()
    executeCode = nil
}

What happened here? Setting a function to nil?! Well, this is a closure in Swift, and a closure is basically a pointer to a function. By putting the statements this way, we avoid any branching, while ensuring the code is still executed once… Given that all call sites remember to set the closure to nil!! How annoying, I though we were done.

The Abstraction

private struct PerformOnce {

    private var operation: (() -> ())?

    init(_ operation: @escaping () -> ()) {
        self.operation = operation
    }

    mutating func execute() {
        operation?()
        operation = nil
    }
}

Finally, the simple abstraction to solve this problem, and hide as much of the unnecessary low-level complexity we don’t care about within another class we don’t have to worry about too much, as long as it’s heavily unit tested.

With the code above, we can now do this:

let doSomethingOnce = PerformOnce {
    // ...
}

doSomethingOnce.execute()
doSomethingOnce.execute()
doSomethingOnce.execute()

Even though there are 3 calls, the code will be executed once. No branching, no nil setting, nothing! Just a very explicit and expressive variable name that indicates the code will be executed once, and indeed that’s what happens.

Ready, Set, Hsssss!

2018-08-18T14:43:09+00:00

Introduction

I was waken up at 3:30 am by the sound of the fire alarm … I tried to ignore it, but that deemed a fruitless effort. So, I moved to the living room, tried my luck there. No dice. So? Screw it. Go to the nearest Costa, have some coffee, and write a game!

Actually, that’s what I’ve been contemplating since yesterday, to write a game with a clean architecture. I have never tried to write a game after acquiring the knowledge from Game Coding Complete book about game architectures.

A Game With a View

While “porting” my Dama game from Cocos to Unity, my 7 year old design couldn’t hold. I kept running into bugs and issues, and sought a better architecture. I don’t recall writing much about “Game View” approach, and doing a quick search on my blog shows only a hit at this design within the devlog.

Basically, “Game Views” is a concept I borrowed from Game Coding Complete book, and it radically improved my Dama King implementation. The idea is simple:

A game view is both a driver + consumer of the game logic.
AI agent, remote player, and even local game graphics are all views.
- AI agent consumes new turn event, and calculates it’s move.
- Remote player consumes a new turn by sending it over the wire.
- Local game graphics represent all the visuals of these events.

New Game?

Why go with a complete new game?

Well, Dama implementation is riddled with legacy decisions, and that will just take too much time and effort to unravel. Instead, to write a simple game, open-source it even, would prove to be a much better learning experience.

Get Down to It

4:56 AM Open Xcode, Create a new project I think it’s best to write this in Swift, and make it a SpriteKit Game.

5:02 AM Exploring the SpriteKit template I haven’t touched SpriteKit since 2015 … It sure has changed.

5:03 AM High-level Architecture Regardless, let’s plow ahead. At a high-level, I would like to make a game that requires both an AI and a multiplayer aspect. This means, at least a 3 player game… Snake (the game I originally planned to write) won’t really do in that case…

5:08 AM Struggling to find the right game… What should it be? … It should be a card game. A card game we call “War” where I come from. Basically, you should claim as many cards as possible, and to do so, you need …

5:10 AM I’m making Swake after all! Screw it, I’m making Swake. It will just have a single player mode, but that doesn’t mean I can’t leverage the Game View architecture!

5:12 AM Starting with the Game Logic So, I’m starting with the game logic… What is the game logic of a snake game?

Snake representation:
- Which blocks it covers
- Which direction it’s headed
Food representation:
- Which block it covers
Game Rules:
- Snake moves towards its set direction by one unit every turn
- Snake consumed food -> Grows by one unit
- Snake collided with walls/itself -> Game Over

5:37 AM Game Logic in place? I now have this simple game logic representation in place … I would like to get a simple graphics representation out of it next, and then a game loop to make it tick:

I removed a bunch of code, as this will hopefully be published in github anyway

final class GameLogic {

    private struct Const {
        static let size = Size(width: 10, height: 20)
    }

    private var state: GameState

    init() {
        state = GameState()
        state.food.position = generateFoodPosition()
    }

    private func generateFoodPosition() -> Point { ... }
    private func displaceSnakeOrGrow() { ... }
}

// Game State:

struct GameState {

    struct Snake {
        var points: [Point] = [...]
        var direction: Direction = .east
    }

    struct Food {
        var position = Point(x: -1, y: -1)
    }

    var snake = Snake()
    var food = Food()
}

6:00 AM Crap is happening on the screen! So, I just went with the most rough implementation, and just updated the GameScene class which was part of the template to be:

override func update(_ currentTime: TimeInterval) {
    gameLogic.tick()
    gameView.tick(state: gameLogic.state)
}

… and then, proceeded to implement the game view itself, as follows:

final class GameGraphicalView: GameView {

    var scene: SKScene!

    private var snake: [SKNode] = []
    private var food = SKNode()


    func tick(state: GameState) {
        cleanUp()
        prepareNodes(state: state)
        positionNodes(state: state)
    }

    private func cleanUp() { ... }
    private func prepareNodes(state: GameState) { ... }
    private func positionNodes(state: GameState) { ... }
}

This is extremely rough because I don’t do any kind of optimization nor animation. I simply remove all the objects, and then add them back again based on the new game state. It’s essentially recreating the whole scene, every single time.

Ideally, we should simply generate “events” from the game logic, perhaps for another day…

For now, this is good enough, and I need to go check if my apartment is on fire or something…

10:00 AM Insomnia No matter how hard I try to sleep, it’s not possible … Also, the my allergies made me snap, as I literally can’t keep my eyes open for 2 seconds straight… So, it’s time to get your maid on.

12:52 PM A fresh start Placebo or not, my allergies are significantly better, and it’s time to get back to it! … First thing I want to do is implement controls!

1:08 PM Navigate away, Cap’! Navigation is working, after a little bit or trouble converting input touches to the appropriate coordinate space, and doing the calculation based on that:

func processTouch(at point: CGPoint) {

    let topThird = view!.frame.height / 3
    let bottomThird = 2 * view!.frame.height / 3
    let centerX = view!.frame.midX

    let direction: Direction
    switch (point.x, point.y) {
    case (_, ...topThird):
        direction = .south
    case (_, bottomThird...):
        direction = .north
    case (...centerX, _):
        direction = .west
    case (centerX..., _):
        direction = .east
    default:
        fatalError("can't happen")
    }

    gameLogic.directionEvent(direction)
}

Now, I would like to actually see the food, and a proper snake! Currently, it looks quite crappy, and the food is indistinguishable.

1:25 PM Things are happening! So, now I have a red snake moving about at a reasonable pace, and I can control its direction using “touch” controls! … I just tried to run it on my device, but I’ve updated to iOS 12, and don’t have Xcode 10 installed… Dang it!

Anyways, so the biggest issue this time around was slowing the run loop. What happened was, after I applied the proper displacement calculation for the graphics based on the game logic representation of where the snake is, the snake would fly off the screen within a second.

I initially thought this was a graphics problem, so I applied a slower loop on the graphical game view, using the currentTime to calculate the time delta, but that just made the game “choppy”. I then realized, the game logic was still running at full speed! So, instead of applying the update throttle on the graphical view, I applied it all the way at the game scene, for both the game logic and the graphics.

This is not ideal, as the graphics should update at 60 FPS! Only the game logic needs to have it’s run loop slowed down, as to control the speed of the snake and such… But meh.

Besides that, now I have a snake that grows as it eats stuff! Time to tackle two remaining concepts: Game Bounds + Game Over.

2:02 PM That took longer than expected… Game bounds was very tricky to get right. First, I couldn’t figure out if the border I was drawing was at all correct or not. Turns out it is, but the translation of GameLogic’s point to scene space was wrong, as it required some calculations and offset.

Even with that out the way, I couldn’t find anchorPoint property on SKShapeNode, which led to a lot of manual work to tweak the Point -> Scene coordinate conversion formula. Eventually, managed to get it done.

So .. Will slow the app down a bit, add game over logic, and call it a day.

2:27 PM Last few touches After adding a game status enum to indicate whether the game ended or not, I actually found a lot of flaws in the implementation. I wasn’t resetting the user input buffer variable, I wasn’t cleaning the snake SKNodes properly as well…

I had to take it across the finish line, so I went ahead and showed a game over label when the game ends, and then automatically restarts on any tap.

Further Works

Even though this was a quick and dirty experimentation with Game View design, it proved worthwhile. What I would like to do with this is implement less dependency between GameLogic and the graphical representation, where they have their own run loops, and they communicate through events rather than the current polling/boil-the-ocean approach.

Here’s da repo!

https://github.com/Mazyod/Swake

Problem Solving

2018-07-22T09:04:11+00:00

Introduction

I wanted to quickly write a python script that would solve a relatively simple problem. The problem is, I needed to generate a plist file based on data defined in a Swift file.

This will involve file access, regular expressions, and file templates. Sounded a bit daunting at first, especially since I don’t have too much time to spend on this task, but I decided to do it anyway, and prove that by breaking down the problem, it shall be quite simple to resolve.

Divide & Conquer

Thankfully, there was some solution in place to solve this problem using Sourcery (a tool used generally for Swift code generation, not plists), however, the main reason I decided to move away from it was the lack of regex support. I mean, what kind of file processing tool doesn’t ship with regex support? Disappointing…

Anyhow… So, with a solution in place, I didn’t have to write the plist file template from scratch, nor did I have to deal with figuring out all the file paths. I took them from the previous solution, and was ready to start!

import os

tweaks_file = os.path.join(os.environ["SRCROOT"], "..", ...)
plist_template = os.path.join(os.environ["SRCROOT"], "sourcery", ...)
plist_output = os.path.join(os.environ["SRCROOT"], "..", ...)

OK, so the boilerplate is in place, what now?

I was writing this script directly into the Xcode build phases field, lol. So, I didn’t have the luxury of an IDE or anything like that, hence I wanted to be able to quickly iterate and find bugs immediately. So, I first wrote how I expected the solution to look like:

def parse_mixpanel_tweak_ids(tweaks_content):
    return ["a", "b", "c"]

def extract_plist_templates(template):
    return "<file>{tweaks}</file>", "<tweak>{display_name} - {key_id}</tweak>"

def generate_tweaks_xml(tweak_ids, tweak_template):
    xml = ""
    for tid in tweak_ids:
        xml += tweak_template.format(display_name="not yet..", key_id=tid)
    return xml


with open(tweaks_file) as f:
    tweaks_content = f.read()

mixpanel_tweak_ids = parse_mixpanel_tweak_ids(tweaks_content)

with open(plist_template) as f:
    template = f.read()

file_template, tweak_template = extract_plist_templates(template)
all_tweaks_xml = generate_tweaks_xml(mixpanel_tweak_ids, tweak_template)
final_content = file_template.format(tweaks=all_tweaks_xml)

with open(plist_output, "w+") as f:
    f.write(final_content)

That’s it! I wrote how I expected the solution to work at a high-level, and them implemented the required functions as dummies in order to iterate on this first. Surely enough, in the code above, I found bugs due to typos, resolved them quickly, and the code was already working great!

Next was the toughest part, which wasn’t too bad, and that is to implement the actual functions to do what they are suppose to.

I’ll save you the boring details, and just say all it took was some regex magic. Again, I didn’t write this 100% correct from the get go, but due to the iteration approach, I quickly know where the bug was introduced, and can easily look to resolve it:

def parse_mixpanel_tweak_ids(tweaks_content):
    regex = re.compile(r"(?:let|var)\s*(.*?)\s*=\s*BoolTweak\(name:\s*\"(.*?)\"")
    return regex.findall(tweaks_content)

def extract_plist_templates(template):
    regex = re.compile(r"%template%(.*?)%endtemplate%", re.DOTALL)
    file_template = regex.sub("{tweaks}", template)
    tweak_template = regex.findall(template)[0]
    return file_template, tweak_template

The generate_tweaks_xml I didn’t even have to change, since it was simple enough to work based on the initial implementation. The two functions above were quite heavy on the regex, and I was rusty … So, I scratched my head quite a while to remember re.DOTALL was needed to match dots to newlines.

Conclusion

The code above is written in a “functional programming” way, since the functions’ outputs are only influenced by the input, and would give consistent results for the same input, no matter how many times they are called.

This way of programming simplifies your life, leads to less bugs, and much more maintainable code. If I wanted to write tests for this as well, it would also be a breeze, for that matter.

Snakes, Everywhere!

2018-04-13T19:29:23+00:00

Introduction

Ehm, for this one, I am really disappointed at my network of pythonistas. I was looking for such critical tool that would change my life forever when it comes to developing Python applications, yet no one could point to the ultimate solution I ended up finding by chance.

The problem statement goes:

I have a Python app I would like to distribute to my teammates
The app is written in Python 3 with many dependencies
My team size is 20+, and not everyone is well-versed with Python
More over, some have a broken Python environment!

So … I need a way to distribute a Python application, absolutely hassle-free. What are my options?

Eggs and Wheels

The most common solution I found online, and people proposed to me, is PyPi. Package your application as an Egg or Wheel, and use a distribution platform like PyPi.

Why Not

The reason this is bad is mainly because it still requires pip and a proper python environment. Even if we assume that’s easily conquered with virtualenv, still dealing with the distribution is a pain. My app has some secrets I don’t want to get out, so … Dealing with a private source is a pain.

Homebrew

I have another inclination, which I was so close to doing. Distribute through Homebrew. I had released a Homebrew formula before (truck), so it was not that bad of an option.

I started to look at the homebrew docs for defaults, and sure enough, homebrew runs your app in an isolated virtualenv, so you don’t need to worry about the users’ mess.

Why Not

I ran into the same issue of private distribution. I thought I could solve that by giving out the secrets separately, and loading them using additional arguments. That is a legit solution.

Then I realized, I also wanted to distribute a webapp alongside the Python server! How would I deal with the static file distribution? It’s probably solvable, but didn’t seem to me like a worthwhile endeavor.

Docker

Overkill.

PyInstaller

😍 My new favorite thing in Python. PyInstaller is an absolute kickass tool for distributing Python apps. All you do is point to your main script, and PyInstaller does the rest of the work.

When configured properly, it can spit out a singl, stand-alone binary executable you can ship as a CLI app. That’s just grand. All I had to do next was a simple shell Mac OS Cocoa application that packages the binary alongside the required resources, and BAM. You now even have some GUI for your Python application, and it is distributed as a Mac OS app.

Next Steps

With the power of a Mac OS app, I intend to provide a slick UX for interacting with the underlying Python app. Things like drag & drop your configuration files, as well as some logging and helper actions to tame the Python.

Conclusion

This post is really intended to take an overview about the available options for distributing Python apps, and doesn’t go into details on how to actually use PyInstaller effectively. That’s for another post....

Pick Your Container

2018-04-03T19:30:43+00:00

Dancing with the Snake

I’ve been engrossed in Python development lately, since we have requirements to build a unified platform for UI testing tasks, such as publishing results, browsing screenshots, … etc.

Being a “seasoned” Python developer, I didn’t think twice about going with Flask, so everything was build on top of that. However, after proceeding to add more UI tests, I ran into such an “unforgivable” error .. Flask wasn’t detecting the HTTP method of the request properly :S

A Leaky Flask

For whatever reason, when our app would send two consecutive requests, where the first request has a JSON body, the second request is improperly handled by the flask dev server .. It show something like:

"{"foo": "bar"}GET /api/path" 405

As you can see, the http method resolved contains the string from the previous request body, which then causes a method not allowed mismatch. I was in utter disbelief, and was certain it was due to HTTP pipelining or some other optimization on the client…

After digging deep and wide in the iOS code, no avail. Everything checks out fine, and Charles shows all requests properly .. It’s time to get inventive.

A Speedy Pivot

No idea how it hit me, but I recalled all of a sudden “bottle”. The python micro-framework that works in an almost identical fashion as Flask .. It was gonna be time consuming, but with nothing else in mind, I moved forward with the migration.

After learning bottle basics, I was able to migrate the basic APIs with super minimal changes, and got the server up and running. That took maybe 15 minutes at most. Once I did that, I ran the UI tests again, and surely enough, the http method bug was resolved!! Yatta!

With this confidence boost, I proceeded to migrate the rest of the server parts to bottle, and completely nuked Flask. I guess that took about an hour or so… I’m now a very happy bottle user, and it is worth mentioning, bottle is WAY faster in launching and processing requests, for my case at least.

Migrating from Flask to Bottle

Here are my learnings for migrating from Flask to bottle:

-import flask
-from werkzeug.http import Headers
+import bottle

 [ ... ]

+# Remove "hop-by-hop" headers (as defined by RFC2613, Section 13)
+# since they are not allowed by the WSGI standard.
+FILTER_HEADERS = [
+    'Connection',
+    'Keep-Alive',
+    'Proxy-Authenticate',
+    'Proxy-Authorization',
+    'TE',
+    'Trailers',
+    'Transfer-Encoding',
+    'Upgrade',
+]
+
+PROJECT_DIR = os.path.dirname(__file__)
+WEB_ROOT = os.path.join(PROJECT_DIR, "snap-audit", "build")
+STATIC_ROOT = os.path.join(WEB_ROOT, "static")
+
+

 [ ... ]

 # Server
 #

-app = flask.Flask(__name__, static_folder='snap-audit/build')
-

 [ ... ]

 # global handlers

-@app.errorhandler(400)
+@bottle.error(400)
 def bad_request(e):
-    return flask.jsonify(error=400, text=str(e)), 400
+    return dict(error=400, text=str(e))


-@app.errorhandler(404)
+@bottle.error(404)
 def page_not_found(e):
-    return flask.jsonify(error=404, text=str(e)), 404
+    return dict(error=404, text=str(e))


-@app.route('/')
+@bottle.get('/')
 def root_path():
-    return flask.jsonify(status="ok")
+    return {"status": "ok"}


 # mock-server-api

-@app.route('/mock-server-api/mount', methods=["GET", "POST"])
+@bottle.route('/mock-server-api/mount', method="ANY")
 def mock_server_mount():
     """Mounts a given postman group
     Expected Query: ?group={group_name}
     """
-    group = flask.request.args.get("group")
+    group = bottle.request.query.group
     if not group:
-        flask.abort(400, description="missing query parameter 'group'")
+        bottle.abort(400, "missing query parameter 'group'")

     if not mounting_service.push_group(group):
-        flask.abort(400, description="group not found")
+        bottle.abort(400, "group not found")

-    return flask.jsonify(status="ok")
+    return {"status": "ok"}


-@app.route('/mock-server-api/unmount', methods=["GET", "POST"])
+@bottle.route('/mock-server-api/unmount', method="ANY")
 def mock_server_unmount():
     """Unmounts a given postman group, otherwise unmounts all groups.
     Expected Query: ?group={group_name}
     """
-    group = flask.request.args.get("group")
+    group = bottle.request.query.group
     if group:
         mounting_service.unmount_group(group)
     else:
         mounting_service.unmount_all_groups()

-    return flask.jsonify(status="ok")
+    return {"status": "ok"}


 # mock-server

-@app.route('/mock-server/<path:mock_path>', methods=["GET", "POST", "PUT", "PATCH", "DELETE"])
+@bottle.route('/mock-server/<mock_path:re:.+>', method="ANY")
 def mock_server(mock_path):
     """Serves the designated mock postman collection"""
     item = mounting_service.item_for(mock_path)
     if not item:
-        flask.abort(404)
+        bottle.abort(404)

     # TODO - perhaps find required example?
     response = item.response[0]
     jstring = response["body"]
     status = response["code"]
     # TODO - enable gzip encoding
-    ignored_headers = ["Content-Encoding"]
-    headers = [h for h in response["header"] if h["key"] not in ignored_headers]
+    headers = [h for h in response["header"] if h["key"] not in FILTER_HEADERS]
+
+    bottle.response.status = status
+    bottle.response.content_type = "application/json"
+
+    for h in headers:
+        bottle.response.set_header(h["key"], h["value"])

-    return app.response_class(response=jstring,
-                              status=status,
-                              headers=Headers([(h["key"], h["value"]) for h in headers]),
-                              mimetype="application/json")
+    return jstring


 # postman-dump

-@app.route('/postman-dump', methods=["POST"])
+@bottle.post('/postman-dump')
 def postman_dump():
     """Postman recording API. Incoming request will be captured into a postman collection
     Expected format:
@@ -295,52 +313,64 @@ def postman_dump():
       }
     }
     """
-    capture_service.add(flask.request.json)
-    return flask.jsonify(status="ok")
+    capture_service.add(bottle.request.json)
+    return {"status": "ok"}


 # snap-audit

-@app.route('/snap-audit', methods=["GET"])
+@bottle.get('/snap-audit')
+@bottle.get('/index.html')
 def serve_snap_audit():
     """snap audit is our React.js app"""
-    return flask.send_from_directory("snap-audit/build", "index.html")
+    return bottle.static_file("index.html", WEB_ROOT)
+
+
+@bottle.get('/manifest.json')
+def serve_snap_audit():
+    return bottle.static_file("manifest.json", WEB_ROOT)
+
+
+@bottle.get('/service-worker.js')
+def serve_snap_audit():
+    return bottle.static_file("service-worker.js", WEB_ROOT)


-@app.route('/static/<path:path>')
+@bottle.route('/static/<path:re:.+>', method="ANY")
 def serve_static_asset(path):
-    return flask.send_from_directory('snap-audit/build/static', path)
+    return bottle.static_file(path, STATIC_ROOT)


-@app.route('/snap-audit/test-result', methods=["POST"])
+@bottle.route('/snap-audit/test-result', method="ANY")
 def record_test_result():
     """Records the given test result as part of the test metadata
     Expected Header: Content-Type: application/json
     Expected Query: ?path=feature/variant/environment
     Expected Body: {"success": bool}
     """
-    path = flask.request.args.get("path")
+    path = bottle.request.query.path
     if not path:
-        flask.abort(400, description="please provide path query item")
+        bottle.abort(400, "please provide path query item")

-    firebase.upload_test_result(path, flask.request.json)
-    return flask.jsonify(status="ok")
+    firebase.upload_test_result(path, bottle.request.json)
+    return {"status": "ok"}


-@app.route('/snap-audit/screenshot/upload', methods=["POST", "PUT"])
+@bottle.route('/snap-audit/screenshot/upload', method="ANY")
 def upload_screenshot():
     """Uploads the given image asynchronously to our storage service. The reason we ask for arbitrary path variable is
     to allow clients to decide their preferred way to structure their screenshots.
     Expected Header: Content-Type: image/png
     Expected Query: ?path=feature/variant/environment&index=3"""
-    path = flask.request.args.get("path")
-    index = flask.request.args.get("index")
+    path = bottle.request.query.path
+    index = bottle.request.query.index
     if not path or not index:
-        flask.abort(400, description="please provide path and index query items")
+        bottle.abort(400, "please provide path and index query items")

-    firebase.upload_screenshot(path, index, flask.request.data)
-    return flask.jsonify(status="ok")
+    firebase.upload_screenshot(path, index, bottle.request.body.read())
+    return {"status": "ok"}


 if __name__ == '__main__':
-    app.run(host='0.0.0.0', port=5858)
+    bottle.debug(True)
+    bottle.run(host='0.0.0.0', port=5858)
diff --git a/requirements.txt b/requirements.txt
index be2a752..b78e1e9 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,9 +1,8 @@
+bottle==0.12.13
 cachetools==2.0.1
 certifi==2018.1.18
 chardet==3.0.4
-click==6.7
 firebase-admin==2.9.1
-Flask==0.12.2
 google-api-core==1.1.0
 google-auth==1.4.1
 google-cloud-core==0.28.1
@@ -12,9 +11,6 @@ google-cloud-storage==1.8.0
 google-resumable-media==0.3.1
 googleapis-common-protos==1.5.3
 idna==2.6
-itsdangerous==0.24
-Jinja2==2.10
-MarkupSafe==1.0
 protobuf==3.5.2.post1
 pyasn1==0.4.2
 pyasn1-modules==0.2.1
@@ -23,4 +19,3 @@ requests==2.18.4
 rsa==3.4.2
 six==1.11.0
 urllib3==1.22
-Werkzeug==0.14.1
-- 
2.15.1 (Apple Git-101)

Ecto.Multi

2018-03-29T02:46:53+00:00

Introduction

I have been ignoring errors coming in from my backend for a while now .. There is only one serious and frequent error, actually, and I’ve no idea how it’s happening, so decided to ignore it .. That was a grave mistake.

Today, a user reached out and complained about his purchases not reflecting on his account! That’s quite serious, and needed to be dealt with ASAP…

Debugging

First, I looked at all the logs I had, and that’s a really annoying chore … I have logs on Papertrail, heroku, custom logging server, and sentry. Yeah .. I was paranoid about the time where I would need to debug purchases, and indeed, that time has come.

The best thing was, the logged exception on sentry had the exact payload sent from the client, hence I was able to replay the request, and get the same error. Perfect! Reproducibility, check!

Now, what is the error? Actually, elixir/phoenix apps default to 8 frames on the stacktrace, so all I could see was:

Elixir.DBConnection.ConnectionError: transaction rolling back
  File "lib/db_connection.ex", line 1523, in DBConnection.fetch_info/1
  File "lib/db_connection.ex", line 738, in DBConnection.run/3
  File "lib/db_connection.ex", line 636, in DBConnection.execute/4
  File "lib/ecto/adapters/postgres/connection.ex", line 105, in Ecto.Adapters.Postgres.Connection.execute/4
  File "lib/ecto/adapters/sql.ex", line 243, in Ecto.Adapters.SQL.sql_call/6
  File "lib/ecto/adapters/sql.ex", line 441, in Ecto.Adapters.SQL.execute_or_reset/7
  File "lib/ecto/repo/queryable.ex", line 130, in Ecto.Repo.Queryable.execute/5
  File "lib/ecto/repo/queryable.ex", line 35, in Ecto.Repo.Queryable.all/4

Dang it!! But you know what the beauty was? I had the foresight to split my backend into micro-services :D .. The affected server was hosted on Heroku, and publishing a new version was as simple as pushing a new commit to Github.

Hence, I configured phoenix to show more stackes, and pushed a new release:

config :phoenix, :stacktrace_depth, 20

The Culprit

After seeing the 20 line stacktrace, even that didn’t make sense, lol, but it at least confirmed my suspicion of which function the error was being thrown.

After looking deep and hard at the function, I was finally able to see it .. it was there, as clear as day… The function, which was part of an Ecto.Multi was handling a Repo.insert with a with statement that “suppressed” a certain DB error that detected duplicate purchase receipts… You can’t do that.

You see, if a DB error is thrown during a transaction, the whole transaction is marked as invalid, and you should rollback before you can start querying the db again. The rollback for Ecto.Multi happens at the end, so I couldn’t suppress this error, unfortunately.

The Solution

Even being away from this code for a freakin year now, I wrote a simple solution that first enumerates the purchases, and filters out duplicates found in the DB. The tests passed from the first try… I was in disbelief.

I changed the function around to break the tests, truly enough, the failed as intended. I kept changing things around just to make sure I’m not missing anything, and nope .. nothing was missing at all. The code simply works. Elixir is the freakin best.

Mining, ethOS and Bots!

2018-03-24T22:33:57+00:00

Introduction

I’ve been into mining Crypto-currencies recently (I’m not sure what’s the proper way to type that, so will refer to them as cryptos from now on .. with the added bonus of triggering cryptography nerds :troll:).

So, what’s the big deal? I’m actually quite late to hop on the bandwagon to be honest. People have been mining ever since the inception of Bitcoin, and that was way back when … The deal is, I’ve applied some python goodness to the mining operations.

Since I use ethOS on my mining rigs, it has been an extremely pleasurable experience configuring and monitoring the rigs. The best part of all this is that the rigs are a source of income rather than another unjustified subscription to satisfy my unquenchable geekness.

Without further ado, let’s breaking this story down. The post is covering these main points: - What is mining? - How’s ethOS working for far? - Solving the monitoring problem

Solving Math Problems

Mining is actually a really clever pun invented by the crypto lords to describe the process of earning cryptos. Gold, even today, is considered to be a valuable material due to its scarcity and defined availability. You can’t synthesize gold out of thin air, and once all the gold on the plant has been mined, that’s it .. People can only circulate the existing gold.

Now, given that you, me, Alice, bob, or anyone for that matter can try their luck and go try mining for gold, and finding gold through such means is fair game, one could say that is a possible way to try and earn a living or perhaps be rich.

Given that definition above, the exact same principle applies to Cryptos. Through an ingenious system developed on top of the blockchain technology, humans were able to somehow come up with a software system that functions in the same way as gold! The main catch here is, literally anyone can define their own Crypto, as opposed to no one being able to define their own “gold”.

So, with that said, the process of mining Crypto, not surprisingly, does not involve digging a mine or any physical activity. After all, the whole system is software-based! How does one mine Cryptos then?

Well, without getting into the hairy details of Cryptos, let’s just say that Cryptos’ reward system is based on lottery, where the more computing power you have gives you a better chance of winning the lottery. (In retrospect, having more gold mining operations means you have better chances of finding more gold).

I am sure you have lots of questions about this, but let’s just leave this here, and move on the to the next topic.

The Mighty ethOS

ethOS is by far the easiest way to get a mining machine up and running (for me at least). Alternatives I’ve tinkered with were simplemining.net and the dreaded Windows 10. Both were surprisingly painful to deal with, and had extremely limited support online in terms of identifying similar issues we were facing.

I mean, think about it. Windows 10 is such a bloated operating system relative to most Linux distros, and everyone has their variation of Windows installation, with a different permutation of software and drivers installed. Hence, it seems people are not able to debug issues properly and detect problems with their setup.

Besides that, it’s not as programmer friendly as a Linux distro, such as ethOS. I haven’t done it yet, but spinning up a crontab with custom scripts to perform my biddings is easy-peasy, lemon squeezy.

Finally, also in contrast with the other options I’ve tried, ethOS simply has all the drivers and scripts you need built-in. I’ve had to run some extra commands to make MSI z270-A PRO and GTX 1060 Rev2 6GB work, but those steps were all properly documented on the ethOS online knowledge base.

To be quite frank tho, some crap was quite annoying to get your head around, especially the config stuff. I used local config for no reason initially, then realized I can just just use gist.github.com for that. Also, there is configmaker thingie advertised by ethOS, but I really want a private, secure config, so … gist was the way to go for me.

P.S. ethOS has built in TeamViewer support, and it is truly kickass.

Better Than Slack

Slack has been my go-to platform for receiving alerts and such .. up until I found out how easy it was to write a Telegram bot. Instead of spinning up a webhook, and dealing with all the setup, which has some very rigid and annoying limitations, Telegram opens a world of possibilities.

So, the story goes … After setting up the rigs, sometimes the rigs would malfunction, and a GPU or two or all would stop hashing. I would like to know about that, so … I noticed that the ethOS panel website provides a JSON api. Hence, why not monitor the API, and trigger me some alerts? Alerts on where?

Given that Telegram is the platform of Crypto, might as well just do it there. It would be a nice exercise as well to finally write me own Telegram Bot. And so, ethogram was born.

With ethogram, we were able to monitor our rigs from within the Telegram group, and given the bot was written in Python, it was absolutely trivial to write various alerts on different metrics. It took me some time to come up with an optimal design for the classes and such, especially when I decided to add unit tests, but since then, I think it is at a perfect place… And now, I want to port it to Elixir xD

Honestly, I’ve been contemplating moving to Elixir since I want to write a more reliable, extensible platform that doesn’t just monitor ethOS, but also tracks Crypto prices, mining pools, profitability, … etc.

I don’t think I’m gonna be doing that anytime soon, if at all, tho … I need to focus on finalizing the UI Testing project I’ve been committed to for the past 5 months or so first… It’s annoying to have such a project lingering around for so long!! Put it’s at a pretty kickass state right now … Once that’s over, I would like to switch my attention to A.I. for a bit, so look forward to reading about that soon!!

Hello... Again.

2018-03-24T21:33:16+00:00

Just Type

I literally just want to breathe some life into the blog again .. I am not sure what I want to write about, but I do want to address the fact that I haven’t written anything for so long, and I really miss this part of my life.

Well, although this is supposedly a dev diary, every now and then I talk about life.. The life of a dev is weird, and is intertwined with his profession…

Unless you are an extremely … unique type of person, it is extremely hard to just sit in front of a computer and just crank out code. Why? I really don’t know, and would really like to find out… Why can’t I just sit in front of the computer for a month, and just type … type away at all these projects I would like to finish …

I’m pretty sure I’ve touched upon this in the past, mentioning it might be the fear of failure. But … what failure? Every programming project, shipped or not, is an addition to your arsenal of knowledge. Programming knowledge doesn’t die with a failed project .. In contrast, it actually becomes just a firm part of your thought process, as that pain of failure warns you about the mishaps that had occurred.

Too philosophical, I know … I’ve fallen a victim to YouTube during burnout, just as one other fellow developer has expressed. I totally related to his situation, and that is such a relief to be honest. We continuously forget that we don’t face problems alone. It is always true that the problem of any individual, especially in this time and era, is the problem of society.

Lol, that was … weird. Anyway, what I wanted to point out is, if you face a problem, take a step back … Diagnose the problem, and if it is considered by your standards a significant problem, think of all the other people in your shoes facing the exact same problem … Then perhaps set out to solve that.

Beyond Typing

OK .. so the hard part of getting start, and back to blogging is over. I’ve typed a whole bunch of stuff above, and it seems like I am back in business!! Well, might as well kick this off properly with an actually useful blog post.

Tune in for the next post, literally gonna start it as soon as I publish this one.

Growth

2017-08-30T14:44:24+00:00

Introduction

To be honset, I thought I lost it. This flow of inspiration to write. Share an experience, learning or otherwise quite meaningless life story. Somehow, I feel it’s back now. After all this time.

What changed? Hmm .. This could be a separate blog post in of itself! Let’s just say I had a turning point last week, which lead to an extremely wide range of results, as an affect of a single action from someone can really influence you in the most bizzare ways…

Well, this blog post is mostly gonna be about what I’ve been up to at work, and how much growth I’ve went through in a seemingly short period of time..

It’s The Little Things

Seemingly small changes can have a profound impact on productivity. My latest proof is FiraCode.

Starting to use this font has made code much nicer too gaze at. As much as I “hated” operator overloading, it has become rather acceptable with the new symbols this font introduce. ~=, |>, ->, … etc, all these look much more natural and nicer on the eyes as well.

After learning about this font, I was sitting in a coffee shop trying to solve a rather annoying problem. How can I structure the networking library tests, to encourage developers to quickly and easily write tests for their new endpoints?

My immediate inclination was to introduce a custom XCTAssert. That would lead to duplication, since I what I was thinking was something in the lines of:

// asserts each property separately
XCTAssertEndpointURL(endpoint, "http://mazyod.com")
XCTAssertEndpointMethod(endpoint, .post)

However .. I realized I should probably just implement Equatable protocol on Endpoint at this point and assert against expectedEndpoint:

// filling up the endpoint struct is tedious!!
var expectedEndpoint = Endpoint()
expectedEndpoint.api.baseUrl = ...
expectedEndpoint.api.headers = ...
expectedEndpoint.api.modifiers = ...

Yeah, it was as tedious as hell. After taking a look at the Endpoint objects we are building, I realized, as a tester, I want to quickly just verify the end result is somewhat appropriate. So .. I ended up adding a custom EndpointData helper class in the tests target, which works something like this:

// FiraCode shines here!
XCTAssert(endpoint ~= EndpointData(url: "https://testing.com/path?query=yes",
                                   method: .post,
                                   headers: ["assert-this": "ignores rest"],
                                   body: optionalBody)

This network endpoint testing deserves it’s own post, but for the sake of this scope, here is where FiraCode sneaks in. This relationship between the two classes isn’t equality, but we still want to match them … Kinda like regex … Which is usually annotated as ~= in many languages … Hence, this.

Native Reaction

I’ve had to invest a lot of time and work on packing a react native component into an ObjC framework to be shipped with a fully native app. The process is not trivial, and for the uninitiated, borderline impossible.

The way ReactNative generates its projects is extremely annoying, since it creates an Xcode project, that depend on other Xcode projects in the nodule_modules directory. At this point, you’re involving yarn into your Xcode build process, which isn’t neat at all.

Besides that, the team is using create-react-native-app project, which requires ejecting before packaging .. Ejection process is interactive, as of yet. This adds an annoying manual step in the build process.

Besides all the RN stuff, building ObjC frameworks is just super annoying, with all the header tweaking, architecture lipo-ing, and namespacing crap.

I Shall Cast a Spell On You

Last but not least to cover this week … Sourcery! After contributing to Swiftlint, I really loved how they were using sourcery to generate the linux tests counterparts. That involved no extra work from the developer side, and was integrated as part of the of the build process.

I’ve worked many times on generating code before, but this stencil-based templating just make things way easier than writing a custom Python script each and everytime that traverses stuff in certain ways and relies on a format-based templates.

I don’t, and probably won’t, be using Sourcery for the Equatable and other protocols it demos, but certainly for other neat automations, such generating ObjC compatible bridge for generics and other Swift only types.

Conclusion

I am liking my new job, 6 months in now! Hopefully, can focus more in the upcoming days on leveraging automation and cleansing the project further from tedious typing and legacy … What can I say, it’s a hobby for me.

ReactNative as a Framework

2017-08-05T18:46:15+00:00

It has been a while since I have posted anything, and to be honest, I shouldn’t be posting anything, since I’ve got loads of other things to worry about..! But oh well, I have searched the Internets high and wide for tids and bits about packaging ReactNative as an iOS framework, and couldn’t find anything useful…

So, here I am saying that it is indeed possible.

I don’t have the luxury to post a whole tutorial with pics and kitten gifs about the topic, but just enough time to brag- ehm, I mean share some of the details of how it’s possible.

To those of you who catch up quickly, and are just here for reassurance:

Simply, create a Framework target in the generated Xcode project, and make the settings and code identical to the app target already created by RN. Done.

For the rest of us, please follow along as I try to recall all the steps required to perform such a feat.

Packaging RN

So, for the purpose of this post, you might wanna give it a shot on the side, and not directly apply the changes to your main project .. Just in case the Xcode project catches fire or something…

So, react-native init PackagingReact in terminal should give you a nice, clean RN project to experiment with. I’m using the following versions of RN:

react-native 0.47.1
react-native-cli 2.0.1

Simply, open the Xcode project residing in the newly created project, and observe in Xcode, under the targets panel, there should be an Application target PackagingReact, and some other stuff like testing and what not.

We simply need to do new, Target..., and choose dynamic Framework. I have tried for a few minutes to make it work for static library target, but that was horrible since we need to bundle some resources and other dependencies…

With the Framework target created, you can select it now under the targets panel where the application target was, and start replicating the settings from the application target onto the Framework target. As far as I recall, I had to do the following:

Link the 11 static libraries
Copy other linker flags: -ObjC -lc++
Replicate the custom Run script phase to the Framework target
Run react-native bundle ... command to generate main.jsbundle
Add main.jsbundle to the framework target

That’s about it, folks. Once that was done, I simply copied over the ObjC code from the application target, to a new class in the Framework, modified that code a bit to load the main.jsbundle instead of searching for the packager, and voala. Works like a charm.

Final Remarks

I am now gonna test to see if I can make the framework target Swift based (no reason it can’t be). After that, I would like to automate this process, so it becomes a single click deployment of the RN application to an iOS framework (which we can possibly link using Carthage even).

Shipping the Game

2017-02-25T22:41:48+00:00

OMG, I STILL CAN’T GET OVER THE FACT THAT I’VE SHIPPED MY GAME!

Yeah, it’s that crazy .. When you’ve obsessed over a project for 5+ years, and it finally crosses the finish line .. I mean .. It’s probably like losing your leg for 5 years, then waking up one day and it’s there! Then, even though you’re absolutely delighted at first, you keep getting these episodes of pure delight for overcoming such a stigma …

Wasn’t It Fun?

So, you’re probably asking yourself by now.. “Wait, you were building a freaking game! Which you loved! Wasn’t it fun?!?!” … Yeah, sure when I was writing code and designing solutions it was liberating .. But then, most of my off time during work or burnout period, it was just a depressing feeling to think about it.

Don’t believe me? Ask anyone who was foolish enough to ask me, “So, how is the game going?” or “When are you planning to ship?”. One of my friends who did ask me this question regularly, said to his other friend, and I quote: “Everytime I ask [Maz] about the game, I see the expression on his face shift immediately”.

That wasn’t fun. Feeling pressured to ship after so long, not being able to see the finish line, skill level not matching the challenges at hand … Check the image below, and I need not say more.

What Are the Implications?

In 2014, I worked at a startup in Dubai as an iOS developer with a decent team and salary. I’m now working in Dubai as an iOS developer with a decent team and salary. The difference? Before shipping, and after… Perhaps you can’t attribute the difference fully to the shipping of the game, but I sure feel it has lots to do with it.

2014: I brooded over the thought of studying Japanese.
2017: I enrolled before settling in, and already enjoying the classes.
2014: I’ve attempted to buy a car, apartment, and make big decisions, but failed to execute anything.
2017: I’ve got a new job + relocation, car, and apartment in 3 months. Also, I feel prepared to take the biggest step in an adult’s life.
2014: I indulged in mindless refactoring at work with low biz value.
2017: I’m proposing and pushing structural changes from day one.
2014: I lacked the confidence at work, home, and in social gatherings.
2017: I’m much more content with how my relationships turn out.
2014: One small external emotional impact could leave me broken for weeks.
2017: It barely takes more than a few minutes to get over something.. After all, I shipped DAMA! XD

I hope my point became clear with the examples above. I am not really sure if my game was a stigma in the past that pulled me back, but at least I’m certain now that it’s a source of pride and confidence.

So, You’re a Success Now?

Nope. Not yet. And here is where the essence of this whole post comes in:

You must find your contentment within yourself.

Only one of all my friends and family truly came close to understanding what finally shipping my game meant to me. Only one!. It wasn’t like I could share this accomplishment with people, and amplify the outcome, so to speak.

Also, the game itself wasn’t a huge hit. If anything, I’ve only lost sales so far and accumulated running costs from the servers and other services, lol. But that doesn’t matter, either!

This brings us back to the quote above, and that is: I was aware this milestone only meant a tremendous amount to me, and no one else. People didn’t like it? Meh. People thought I wasted 5 years of my life? Meh. What do they know?

Bonus: But You’ve Abandoned the Blog!

Some may point out that the blog was one of my “biggest” accomplishments during my pre-shipping, and it didn’t come back, yet. Actually, this is an excellent point, and it has to do with the actual value this blog had! Try reading the posts, and you’ll realize what a mess 90% of it was.

No proper structure, barely any useful content, and when it was finally there at a good level, it was really just me scraping for any remnants of self-dignity and busywork to make me feel productive so that I could muster some motivation to keep moving forward.

Conclusion

I’m super glad that I’ve seen my game through, and I wouldn’t change anything if I would go back in time. It wasn’t about the actual game; it was about the learnings and the journey.

Let's Encrypt Phoenix!

2016-12-19T18:53:02+00:00

Introduction

After a previous post outlining the importance of using SSL, especially when dealing with websockets, I needed now to implement auto-renewing certificates as a cron job or whatever.

Previously, I had used the “standalone” approach to get my certs, which worked great.. However, it’s not ideal when you want to keep your server running during the renewal process! You see, the standalone server needs to bind to port 80/443, but the running server already occupies those ports.

The Root of Groot

Actually, I wanna talk about Webroot, a certbot plugin that allows you to verify your domain through an existing web server.

So, the way webroot works is it basically writes the authentication files to a path you specify, and then attempts to request that file for the authentication process.

Sounds awesome, right! Just write your files to priv/static, and we’re golden! .. Not. Even though I tested creating files in that directory, then requesting them using curl, and it worked.. This approach didn’t work in production. I believe it has to do with the code reloader.

After much dabbling around with the production server, I finally realized that the actual server is running in the _build folder! Simply changing priv/static to _build/prod/lib/app/priv/static finally did the trick..

The Steps

So, to get webroot working with your phoenix server, first change your cli.ini configuration like so:

rsa-key-size = 4096
email = me@example.com
# disabled for renew: domains = my.domain.com
text = True
authenticator = webroot
webroot-path = /path/to/project/_build/prod/lib/myapp/priv/static

Please take note above of the following:

We disable the domains options since it upsets the renewal process. Uncomment if you are generating for the first time.
Make sure to set your email.

Once you do that, you can proceed with running the certbot program:

./certbot-auto renew --dry-run

Conclusion

I felt inclined to donate to EFF for their fantastic work on let’s encrypt, and so feel inclined to promote them here as well:

Donate to EFF

Useful Unity Assets

2016-12-10T21:49:46+00:00

Introduction

As I wait for my laptop battery to drain so I can do complete power cycle for the battery, might as well write a post!

I recently started digging through the Unity Asset Store, and found some gems of assets! Let’s take a look.

Moderation

Before I go into this, let me say this: Moderation.

Pulling every asset store package to “speed up” your development process would be a fatal mistake. A lot of assets out there will actually do more harm than good. So, let’s first take a look at which assets are OK to use:

Battle-Tested: The asset must have 4.5+ rating, and reviewed by a reasonable number of users, with insightful reviews.
Focused: The asset must solve a specific problem, and does it well, instead of advertising that it solves everything for you.
Docs & Features: Make sure their docs are readable so you can actually use it! Also, it supports the build configurations you need.

With that being said, here is my list sorted from my favorite, all the way down to the least preferred. However, I use them all, so they’re all great!

Json.NET

This one gave me a true piece of mind with so little done on my end. Basically, I was using Newtonsoft.Json in my game, and it was breaking under certain conditions in different build configurations. I would have to spend days testing and making sure I work around all the issues!

Thankfully, I found this neat fellow! It has the exact same namespace and APIs as Newtonsoft.Json, so all I did was nuke the open-source version and import this one from the asset store, and just like that… All errors begone!

I love it the most because it solves a super specific problem, JSON serialization, in a super efficient and straight-forward manner.

I2Localization

As I was losing my mind managing localized strings, trying to solve Arabic RTL issues with NGUI, I stopped everything and started doing some research. How do people deal with this issue?

That’s when I found I2Localization. It boasts RTL support, and with a little bit of a setup, it syncs with Google docs right from the Unity editor! Another super useful feature is statically typed localized strings, using the helper scripts that ship with this asset.

I had some issues with RTL initially, but I quickly added a fix, and submitted a report to their forums. They were fast to respond with confirmation of the bug and shipping a fix soon after.

It’s also kind of unfortunate how they don’t support multi-line RTL 100%, but it works for my current needs, so meh. Ultimately, I would really recommend this for the Google docs syncing and statically typed strings alone!

BestHTTP

This is the last one I want to mention, which ironically, is what I need the most. I don’t love it as much as the ones before, but I needed it so badly, that I have to overlook its issues.

BestHTTP is simply the best HTTP experience you can have in a cross-platform Unity game. I say cross platform because, if it was for one platform, I’d personally prefer to use native networking capabilities instead. Alas, my game is cross platform.

The HTTP request APIs are 100x better than the stupid WWW thingie Unity provides us. The recent UnityWebRequest is even worse in terms of stability! So, I use this library for GET/POST HTTP requests, and it really delivers the thing you need.

The main reason I got it, however, is the websocket feature it has. It’s a well written/maintained websocket implementation, with main thread callbacks, very detailed logging and error handling that helps you crack down the nastiest of issues. The next best thing I found that provides websockets was an outdated websocket-sharp package from 2014, which worked surprisingly OK, but I just did trust it..

The only reason this isn’t on the top of the list is the way it synchronizes callbacks to the main thread. It uses a MonoBehaviour, which uses mutex to check for updates on update! Like, why?! I don’t need to check HTTP results 60 times a second. There should be a better way using semaphores or similar to only trigger updates when needed.

All in all, this library is great, and you should get it if you need serious HTTP functionality or websockets.

Conclusion

My laptop is out of battery! Time to shut it down.

A Sockety Tale

2016-12-10T14:16:51+00:00

Introduction

On a fine Friday evening, after I went over my git repos making sure everything is synced with upstream, I delightfully pack up reminiscing the past week. It was a fine, productive week! Lots of trello cards moved over to the done column.

As my phone’s slack was pinging, alerting me of the status of Unity cloud builds, I got an idea. Why not download a build on my Android and try it out on the way home? The bots server was already running, so they’ll keep me company.

As I started strolling back home, I downloaded the app, installed it, and started using it…

Let the Debugging Begin!

Portable Debugger

To my utter horror, the user data was failing to load! I restarted the app, tried several retry buttons; alas, nothing seemed to work.

At this point, I usually despair at my future demise, as I had to hook up the device to a debugger to figure out what’s going on … However, as part of this productive week, I had hooked up a remote logger!

So, as I am still making my way back home, I launch the online console and immediately welcomed with a wall of logs so granular, they looked like this:

[...]:  [DEBUG]: Logger initialized...
[...]:  [DEBUG]: [HTTPConnection]: 'ws://dama.level3.io:80/socket/websocket?token=...' - Connecting to dama.level3.io:80
[...]:  [INFO]: [HTTPConnection]: Connected to dama.level3.io:80
[...]:  [INFO]: [HTTPRequest]: Sending request: 'GET /socket/websocket?token=... HTTP/1.1'
[...]:  [INFO]: [HTTPRequest]: 'GET /socket/websocket?token=... HTTP/1.1' sent out
[...]:  [DEBUG]: [HTTPConnection]: ws://dama.level3.io:80/socket/websocket?token=... - Receive - protocol: WebSocket
[...]:  [DEBUG]: [HTTPResponse]: 'ws://dama.level3.io:80/socket/websocket?token=...' - Receive. forceReadRawContentLength: '-1', readPayloadData: 'True'
[...]:  [DEBUG]: [HTTPResponse]: 'ws://dama.level3.io:80/socket/websocket?token=...' - Status Line: 'HTTP/1.1'
[...]:  [DEBUG]: [HTTPResponse]: 'ws://dama.level3.io:80/socket/websocket?token=...' - HTTP Version: '1.1'
[...]:  [DEBUG]: [HTTPResponse]: 'ws://dama.level3.io:80/socket/websocket?token=...' - Status Code: '400'
[...]:  [DEBUG]: [HTTPResponse]: 'ws://dama.level3.io:80/socket/websocket?token=...' - Status Message: 'Bad Request'
[...]:  [DEBUG]: [HTTPResponse]: 'ws://dama.level3.io:80/socket/websocket?token=...' - Header - 'Server': 'Cowboy'
[...]:  [DEBUG]: [HTTPResponse]: 'ws://dama.level3.io:80/socket/websocket?token=...' - Header - 'Date': 'Fri, 09 [...]:  [DEBUG]: [HTTPResponse]: 'ws://dama.level3.io:80/socket/websocket?token=...' - Header - 'Content-Length': '0'
[...]:  [DEBUG]: [HTTPResponse]: 'ws://dama.level3.io:80/socket/websocket?token=...' - Header - 'Connection': 'keep-alive'
[...]:  [DEBUG]: [HTTPResponse]: 'ws://dama.level3.io:80/socket/websocket?token=...' - ReadRaw - contentLength: 0
[...]:  [DEBUG]: [HTTPResponse]: 'ws://dama.level3.io:80/socket/websocket?token=...' - ReadPayload Finished!
[...]:  [DEBUG]: [HTTPConnection]: ws://dama.level3.io:80/socket/websocket?token=... - Receive - Finished Successfully!
[...]:  [INFO]: [phx][socket]: Request Finished Successfully, but the server sent an error. Status Code: 400-Bad Request Message: 
[...]:  [INFO]: Websocket connection error: Request Finished Successfully, but the server sent an error. Status Code: 400-Bad Request Message: 
[...]:  [INFO]: Purchaser: OnInitialized - PASS
[...]:  [WARN]: We are not online

As I went through the logs, I found the culprit. For whatever reason, I was getting a 400 Bad Request response from my websocket requests. Thankfully, my backend also logs warns and errors to the same console.

With a few taps, I switch over to the backend logs. Unfortunately, there were no logs whatsoever. That meant, the server wasn’t throwing any warnings or errors…

Dive Into the Logs

I was getting closer to the apartment, so I started thinking, where can I begin to debug this? The server verbose logs of course!

So, as soon as I entered the apartment, I took out my laptop and logged into the server to view the logs files. These were configured to log everything, with log rotation in place, it’s bound to be useful, right? Nope. Even those logs didn’t show me anything of interest about this bad request error!

Am I stuck at this point? Is it the new HTTP library that I switched to recently? .. Wait, this error never showed up on the Unity Editor, why is it happening on Android? The main difference was really the Internet connection. Android was using a data plan…

As I switched my phone over to the apartment WiFi, I prayed silently that the issue would persist and not go away (because how the hell am I suppose to debug an issue with the cellular connection!!). The moment of truth was clear, the issue did not persist! The issue was with a high degree of certainty with the cellular connection.

Sysadmin Toolbelt

Now that I was certain the issue manifests itself while between the client and the server, I had to take a look from the server side as to what’s going on.

I no longer remember when did I learn about tcpdump, but I do remember a developer taught me about this tool on some Slack channel. This is definitely the right tool for debugging this insane issue!

After launching tcpdump for the first time, and seeing it dump thousands of lines of gibberish, I had to first isolate the problem!

First, I shutdown the bots server, since it was still going strong and generating insane amount of traffic! After that, I wanted tcpdump to show me legible data about my specific tcp connection. To do that, I used tcpdump ascii output option, as well as source IP filter option:

tcpdump -A -i eth0 src 31.4.188.x

That did it! As I launched my game and connected from the 3G connection, and then the apartment WiFi, the difference was clear… The headers when connecting through the 3G connection were malformed! Specifically, the Upgrade: websocket header was missing. How can the server process such a request? Obviously, it couldn’t.

Our Savior

As usual, the real solution is found on StackOverflow. I easily searched for the problem and found the answer. My cellular company were using a proxy in the middle, which counts as a hop, which removes the Upgrade: header. The solution? Use secure websockets!

I wanted to secure my server for a while, but it wasn’t critical. This time around, I just had to do it for the sake of allowing users across the variety of connections out there to connect.

Since my server is running on DigitalOcean, it took me less than 10 mins to setup letsencrypt (certbot), generate self-signed certificates, switch phoenix to use https, and push a new build to Unity Cloud Build that uses secure websockets.

The rejoice came after the cloud builder finished, and I was able to install the new build, and confirm that the issue was resolved! Yay!

Conclusion

What I really like about this tale is the range of technical knowhow involved in debugging the issue, including rudimentary steps performed on the phone itself!

Programmers make for great investigators. They live by Sherlock Holmes phrase:

Once you’ve eliminated the impossible, what you’re left with, however improbable, is the truth!

Tech Buffet

2016-12-04T23:45:23+00:00

Introduction

This blog has been side stepped for quite a while .. For some reason, this pattern happens every time.. During the beginning of the year, I’m like “I’ma blog all year!!”. However, I slow down during the middle, and near the end realize, “Oh crap, let me play catch up now”.

All You Can Eat

I don’t think it was a bad year for the blog, though, all things considered. I’ve finally overcome my iOS comfortzone and delved into other interesting fields, like Elixir and Unity.

I believe I also migrated my app backend to Heroku this year, and started using Heroku a lot more in general. I mean, the main restriction that bothers me is the freaking power cycle, but other than that, I just dump all my servers there.

I’ve indulged in docker during the beginning of the year and realized it was a crazy and unnecessary overhead for an army of one. Then, a friend convinced me to try out docker cloud during the end of the year, which was an amazing experience at first .. Until docker cloud “Lost contact with a node”… Seriously? Also their support forums suck. There, I said it.

Let’s see, besides that …

How about using the new Android Studio external native build feature? Did you know Android studio can freaking import a CMakeLists file into gradle and elegantly build that into a native library, packaged as an AAR?! That was neat.

Regarding native libraries, docker wasn’t a complete loss this year. I’ve actually learned about and utilized Dockcross; a set of pre-configured docker images that contain cross-compile toolchains. To build my C lib for linux on my Mac, all I have to do now is ./dockcross ./build.sh. How neat!

Of course, the best thing that happened this year was meeting Elixir and getting to know it better. Just like a steak, you let it cook steadily, while incrementally applying the right set of spices. By the end of the process, you got yourself quite the indulgent!

Tech aside, I was really lucky to have worked with a super-fun team over at Bigfootgames.com. They helped my get the ball rolling with my game. I also had the fortune to knowing two super talented guys; a designer and a composer. Their fine work is the main reason I have the energy to keep at making this damn game (no pun intended).

Let’s bring tech back again, since I’m not nearly done with that yet. This year we witnessed the drastic change that Firebase went through. Luckily enough, I was able to work on two shipped products that utilize their new services. Firebase is just great; you should take a look at it.

While we are in this area, might as well mention the insanely useful papertrail logging service. I’ve been having issues with logging for a while, and never really cracked the problem until I found Papertrail. They have a neat, UDP based remote syslog setup which is compatible with an abundant number of technologies, including Elixir.

Conclusion

This is just a dump, so I do apologize for the lack of structure. I can also summarize my 2016, from the tech side at least, as the year I finally understood Unity.

Regardless of anything else, Unity was this thing I just couldn’t click with. Having committed to learning it, during my escape to Japan, was what will probably last with me the longest. I have arguably spent most of my time in 2016 sleeping, obviously, but second to that, doing Unity related stuff.

Elixir was fun, but not really the domain of my project. It’s the assisting component that I unfortunately had to simply make work, and move on to perfecting the experience in Unity.

So, that took about 15 mins to write, lol. Says a lot about how long I’ve been wanting to dump this stuff… Would be nice to make it all dumps instead…

Unity Ads

2016-11-13T11:14:08+00:00

Unity ads is a great opportunity for games to make some serious revenue from the free players segment. These players would never pay a single dime, so making them watch a video ad results in revenue for your game.

The Unity platform is insanely good with video ads. Basically, you click a few buttons in unity, add Advertisement.Show();, and your done.

This is all nice and dandy until you realize that you need to now reward the user for watching the ad. The ad runs on the client, so it might be compromised .. Unity solves this problem as well by providing Server-to-Server callbacks.

So, basically, once a user watches an ad, Unity servers send a callback to your server with the user id that confirms that action. You can then be sure that the user can be awarded the promised reward without worrying about it being compromised.

The only issue here is that implementing backend callbacks is a chore, and I don’t think enough players will bother hacking the app. On the other hand, adding the reward system within the client is just too risky …

While throwing my thoughts here it occurred to me that I can implement the server callbacks not mainly to prevent hacking, but mainly to make the implementation easier. With the callback sent to my server, I can simply update the profile, and push that update to the client.

on second thought

2016-09-27T15:59:32+00:00

Well, I did need to write about the current authentication design, not that I think it’s amazing or anything like that, but in fact, I need to write it out in order to properly think about it.

So, what’s the deal with the authentication flow?

So, I decided to go with a pretty “novel” approach for authentication, and that is to redirect the user to a website that (s)he can then authenticate through, which “upgrades” the associated account…

So, everytime a user installs the app, a new “guest” account is created, and a device token is associated with that account. The device token, as the name suggests, is a unique auth token for that device, which provides access to the account without the need to enter the username/password everytime. Pretty basic stuff..

Now, once the user logs in on the web portal, as I call it, the device needs to “know” about this authentication process, and change the initial guest account to the new logged in account.

My approach to this problem was to send the guest auth token with url redirect. I am not worried about the security of this token, since it is always a token of the guest account, which isn’t significant.

So, once the user is on the login portal with that token, we can fetch the guest account and prepare for the upgrade process. If the auth process succeeds, there will be a new account we want to “migrate” to…

I am thinking of simply migrating the device token to that account, and once the user is back in the game, we detect the token and that it has been migrated, and send an updated token of the new account, which is then securely stored on the device for further operations.

Everytime I think about it or try to find issues with this approach, I don’t see anything wrong, but also every “new” design you come up with in programming will always make you anxious since if this is so great, why isn’t everyone using it?

Well, I don’t have time to care about that, I need to get done with this…

i forgot about this

2016-09-27T15:55:47+00:00

Holy crap, I completely forgot about this devlog >__< .. definitely should not have…

I don’t remember what the last post was about, but since then, just too many things were worthy of at least dumping here .. Learning new optimization tricks in python thanks to the interview process preparations, Swift 3 changes, Phoenix controllers and HTML forms, and of course, the ordeal with ecto.

Well, I don’t feel like writing anymore for now, but should revisit this once something comes up.

Changeset

2016-08-12T14:59:05+00:00

That was painful … Dealing with Ecto changesets.

The thing is, I’ve been trying to implement a way to use changesets in order to protect certain properties of the model from being modified by an external source. For example, users of the app can change their username, but not their rating.

To achieve something like that, I first created external_changeset on the user model:

def external_changeset(user, params) do
  ...
end

Given the user, and the proposed parameters to be changed, we should return a changeset with the following conditions:

The changes shouldn’t modify protected members
The changes shouldn’t render the model in an invalid state (e.g. empty username)

Well, I thought it was simple at first, because I’ve seen the cast method in action. cast is a method on Ecto.Changeset that allows you to control the changes in a certain way…

The thing is, I’ve always thought that the rules you pass into the cast method are applied to the Ecto.Changeset model itself, but no! The rules are actually applied to the final resulting model! What does that mean?

Well, first, let’s see what the cast method does:

user
|> cast(%{username: "mazyod"}, ~w(username email))

Depending on whether you are on the latest ecto release or not, cast takes the model or changeset as the first parameter, then the changes as a map, then an enumerable of fields that are required or can be modified.

That’s where my earlier statement comes into play. Notice how even though we “require” an email field, the changes map doesn’t have an email. That’s fine, because if the user object itself has the email set, that means the overall changeset is still valid!

Phew! I needed to write all that out, since I am sure my future self will need it.

Doing Yoga with C#

2016-08-08T16:17:02+00:00

Introduction

Throughout my game development journey with Unity + C#, I have found the C# language extremely flexible and versatile. By utilizing the right techniques, one could eliminate lots of boiler plate and achieve remarkable expressiveness.

I talk about expressive code a lot in my posts, but if you need a refresher, expressive code is simply doing more stuff with less code. I would like to shed some light on two use cases today, both of which utilize generics to achieve their expressive nature.

Event System

An event system, observer patter, publish-subscribe .. etc, is a very basic design that is mostly needed in any type of software system. It helps you achieve decoupling between an event publisher and its subscribers.

So, we want to have a nice event system that satisfies the following conditions:

Statically typed
Clear usage
Easy to define new events

Since C# has the event data type, which allows you to define events and start utilizing the observer pattern, let’s evaluate that first:

public delegate void EventCallback();
public static event EventCallback OnAccountLogin;

// publish
if (OnAccountLogin != null) {
    OnAccountLogin();
}
// subscribe
AccountManager.OnAccountLogin += AttemptToConnect;

This is a bit redundant. We have to define the delegate, check for null if we want to trigger the event, and the line isn’t very clear between who is the publisher, and who is the subscriber. Both perform a call on the event itself, after all..

The developer working with me took a slightly simpler approach using the Action class:

public static Action<float> OnTurnTimeUpdated;

public static void TurnTimeUpdated(float time) {
    if (OnTurnTimeUpdated != null) {
        OnTurnTimeUpdated(time);
    }
}

// publish 
UIEvents.TurnTimeUpdated(dt);
// subscribe
UIEvents.onTurnTimeUpdated += TurnTimeUpdateHandler

It gets super annoying after defining a few events, rewriting that null check and extra method each and every single time…

This also has a horrible side effect .. Multiple times, the game crashed because someone was triggering the Action object directly, instead of going through the method!

// instead of doing this...
UIEvents.TurnTimeUpdated(dt);
// they did this...
UIEvents.onTurnTimeUpdated();

The code above illustrates two valid ways to trigger the event, however, the second approach crashes the game if there aren’t any handlers, since it’s missing a null check. This made me seek C# for some yoga lessons and introduce some versatile construct that can solve this issue.

Enter DKEvent

I called the class DKEvent, and this is how you can use it:

public static DKEvent OnAuthStarted = new DKEvent();
// publish
UIEvents.OnAuthStarted.Trigger();
// subscribe
UIEvents.OnAuthStarted.action += AuthHandler;

Finally! No null checks, very clear distinction between publishing and subscribing, and best of all, it’s a single line of code to define new events! How can we achieve this beauty?

Simple enough, by using generics!

public class DKEvent {

    public Action action;

    public void Trigger() {
        if (action != null) {
            action();
        }
    }
}

public class DKEvent<D0> {

    public Action<D0> action;

    public void Trigger(D0 d0) {
        if (action != null) {
            action(d0);
        }
    }
}

...

Things to note about the code above:

You can redefine the same class name, but with different number of generic arguments, as many times as you like.
We have finally encapsulated the null check, and we simply define new events using the code we saw earlier.

Stored Properties

Jumping directly to the second use case, which is stored properties. Imagine you have a class that has properties which need to persisted to disk. Typically, writing such property requires lots of boilerplate code:

public GameParameters parameters {
    get {
        return PersistenceManager.Instance.GetValue<GameParameters>(GameParameters.PersistenceKey, GameParameters.DefaultParamters);
    }
    set {
        PersistenceManager.Instance.Set(GameParameters.PersistenceKey, value);
    }
}

The above code assumes you have a nice generic persistence manager class which you can easily call to store and retrieve properties as needed. This code isn’t too bad, but we can do better.

Using the same approach demonstrated earlier, we can replace the code above with:

public StoredProperty<GameParameters> parameters = new StoredProperty<GameParameters>(
    GameParameters.PersistenceKey, 
    GameParameters.DefaultParamters
);

The code may look ugly, but it’s just a one liner. With a single line, we can define a new stored property, without the need to rewrite the calls to the persistence manager and whatnot. In order to use the new code, however, we have to use the designated accessors:

parameters.Set(newValue);
var params = parameters.Get();

This is because of the way StoredProperty is implemented:

public struct StoredProperty<T> where T: struct {

    private string key;
    private T defaultValue;


    public StoredProperty(string key, T defaultValue) {

        this.defaultValue = defaultValue;
        this.key = key;
    }

    public T Get() {
        return G.App.persistence.GetValue<T>(key, defaultValue);
    }

    public void Set(T val) { 
        G.App.persistence.Set(key, val); 
    }
}

Conclusion

The only reason I know so much about generics is because I’ve played around it a lot with Swift. In swift, you can achieve a hell lot more than what was presented here, thanks to protocol extensions and type inference.

This feels like a rushed post :( .. it probably is… I just really didn’t want one whole month to pass without posting anything!!

Spitting It Out

2016-08-07T18:26:02+00:00

Not a very friendly title, but I really need to throw out there what’s happening these days. It’s too valuable to be lost in translation or in a long term memory archives…

For starters, I finally realized that toon shaders don’t cut it no more.. They are way too complicated for flat design! Besides, the whole idea behind toon shaders is to emphasize the edges and clean the surface of an otherwise rough texture.

So, while looking around the Unity project, I deleted the folder containing the toon shaders by mistake, and decide the heck with it. Let me find a better way to do dis.

Finally, I realize that the “Mobile” category in Unity shaders isn’t for low quality shaders. In fact they are high quality, just optimized for mobile by limiting certain functionality. Over there I found VertexLit shader and Unlit.

Skipping VertexLit, Unlit was all I needed, really. Using the Unlit shader meant that the model texture will be draws AS IS, without any modification whatsoever by lighting, reflection or shadows in the render scene. For my flat game, this is a godsend.

After applying that shader and seeing how great the game started to look, I also decided to do a quick pass over the textures that were added to the game. Changing those as well helped me achieve the absolute perfect look I was aiming for! Yatta! Well.. one nuisance is that the 3D models themselves still look a bit off, but whatever.

The other thing which tempted me to write this post was the discovery of the global namespace in csharp, lol. I was having issues using a global class because I had defined the same name in a local scope… MonoDevelop was an absolute hero by suggesting that there is conflict with global::ClassName. Nice!! So, in order to use the global name, just add global::, and we are good to go..

That wraps it up for now, I need to go back to work! :D

Anonymous Socket

2016-08-04T16:45:04+00:00

Even though I set the title as “Anonymous Socket”, I decided not to go that route after all…

Here is a quick brief of what’s going on inside my head:

I need anonymous clients to connect to the backend and generate a temporary guest account to use in case the user doesn’t want to create an account. In order to create the guest account, I have implemented a simple REST API.

After looking at the implementation on the client side, I realized that using Phoenix channels is way simpler and easier than performing API calls, so I was thinking of creating a new socket endpoint for anonymous clients. A client would connect anonymously, and be given a grace period of 5 minutes or so to perform the actions it needs, such as registration or login.

The second idea sounded great, easy to implement, and in alignment with the rest of the backend .. SHIKASHI!

It would mean I’d have to manage anonymous states and sockets, which can quickly become vague and cumbersome. When using REST APIs, the request/response cycle is all you have to worry about. Once the response is sent back to the client, it’s over.

With that in mind, I figured in the future I’ll have a separate “authoritative” server that would deal with these anonymous pricks and issue secure tokens for them to access the game servers. With that in mind, if I use a REST API, it would be easier to use Heroku or some other service and language to distribute the risk/load across many servers… I’m probably over-thinking this at this point xD

Honestly, the real reason to leave the anonymous socket behind for now is time. I keep thinking about how the anonymous socket implementation is gonna be, when I already have a working implementation of the REST API client and server. So, for the sake of time, let’s stick with the REST API.

…

With that being said, I really need to move on to implementing the functionality required to register/login the user. The API is implemented, but still not hooked up to the UI yet… Should be simple enough to do that.

Once that’s done, I should focus on logout, figure out how that should work, while making sure the user has the ability to recover their account… Finally, I really need to implement social login, for simplicity sake. Even Pokemon Go uses Google sign up, which means it’s the way to go.

…

You know, for the logout stuff, I really have no choice but to add an email field when the user chooses to upgrade their guest account. The user would already be invested, anyway, and it will be for their sake so they don’t lose their account. As for social login, the beauty is that we don’t have to worry about them losing access there.

More Errors

2016-08-04T14:08:23+00:00

Even more errors, and more edge cases to solve … If you are developing a game, really there is never a dull moment around here .. Every single detail is a challenge that requires careful thought and planning. The idea is that once you start executing, everything should just fall into place.

So, back to the issue at hand .. After handling the simple error dialogs part and food, now I reached the game scene, and need to handle the errors there. As highlighted previously, the game view syncs the game scene graphics through an event queue .. So, I could just create a new event for channel errors and call it a day .. But, there are many types of errors that could occur in a game.

First, there is the join error. So, the player somehow reached the game scene, but due to some circumstances, the game is no longer available, so that results in a join error. I think this is simple since I can just enqueue a join error event to the events queue, and it will be handled gracefully. This works well because I actually also have a “game ready” event in place which is only sent to the graphics part if the game has initialized successfully.

Then, there are two other errors related to rematch. Since rematch is a paid action, therefor the request may be rejected due to insufficient funds. Another issue may happen when a rematch is requested, but it goes away before that packet reaches the server.

I think that in either case I can just handle the error as a “Rematch Unavailable” event, which is already in place. That event will disable the rematch button, and show a label “rematch unavailable”.

Hopefully, this will resolve all the channel error handling, for now!

The only missing piece in the game after that is implementing a proper account flow, where users can create accounts, login with existing accounts, logout, and hopefully be able to recover accounts .. I can’t believe I’m saying this .. But once that’s implemented, it’s time for BETA TESTING!!

Errors Localization

2016-08-04T12:01:48+00:00

Forgot that on a similar note to the previous post, I was wondering how I should deal with error message localization. I can deal with the localization on the frontend or the backend. If I choose either, there are also multiple ways to approach either one… Then, there is the hybrid approach, where the backend sends all localized strings, and the client picks the right one…

Ideally, I would probably just tag the locale when opening the socket, assign that to the socket assigns map, and use that throughout the backend to return the appropriate localized string… Actually, I really should do just that.

In elixir, people seem to be using gettext to localize stuff, but I haven’t tested it out yet, hopefully it’ll be simple to use, and would solve this issue for me.

Well, I guess that’s it then! Just attach the locale to the socket when joining.

errors

2016-08-04T10:42:36+00:00

Hmm.. here I am again just typing stuff instead of working on my game .. This is the penalty of inexperience and lack of another brain to bounce ideas off of..

Well, the issue I am facing now is dealing with backend errors on the client side. The backend gracefully handle extreme cases, such as insufficient coins and state unavailability by returning errors to the client …

Now, they are extreme cases because normally, the client app will check if the player does indeed have enough coins or not and warn them without really bothering the backend. It is only if the client data somehow gets out of sync, or if a player thinks themself smart enough to hack the game is when we reach this case.

With that being said, I realize that I don’t really need dokubetsu handling for these cases, especially not for version 1.0 .. I should just roll with simple dialogs and what now, and move on.

The only issue is that I should always handle joining errors, no matter what the reason is, because otherwise I would have a reference to a dead channel and the game state would think it is active. That won’t do at all.

So, first I should rename OnError function to OnJoinError, since that’s more representative of what’s going on. Then, I can handle those errors by resetting the channel somehow… Then, we can have specific error delegate methods per channel, per reply use case. You see, these errors that I am handling here aren’t random. The are specifically created and sent by the backend. So, I’ll just create delegate methods for each, and handle them as appropriate.

Whenever things click so well, my mind just wanders off in the distance on how I can leverage this perfection in different applications… I then need to pull myself back to reality and focus on finishing this crap.

Well, let’s do this and hope it works out!

rants

2016-08-02T15:14:01+00:00

Just needed a quick way to track some tasks, so here it goes…

Found a bunch of crap surrounding the dialog implementation. Most of it was easy to cleanup, however, the animation code I just can’t find it… Looking at how the dialog isn’t deactivated nor cleaned up after being hidden also made me decide to just not cache panels....

Like, I’ve specifically had this talk with the developer, telling him:

There are two hard problems in Computer Science, the first being cache invalidation.

Yet, he insisted “it’s easy to cache”. Sigh. Yeah, you just store the crap in a variable and call it done… Completely ignoring the invalidation part, amazing.

So, let’s clean that up… OK, cache nuked… Running… Works as expected in most of the pleaganels, but the dialog stuff is interesting, since the dialog simply just stays there after the animation, and a new one is instantiated if we click again, leaking the old one… So, need to resolve the dialog flow.

Hmm… cleaned up a few scripts around panel controller, and forgot what the hell I was doing… Oh, dialogs… How are they even managed?

Well, it was pretty neat, actually. There was a script that automatically searched for any tween component in panel you push, and triggers it. Once I searched the prefab for that tween component, it was simple to alter the flow and fix stuff… Still need to figure out why the dialogs aren’t being destroyed…

I finally figured out why it was so hard to read the flow of the code.. It is basically because different things were doing the same thing… I mean, imagine you have a navigation stack. You push stuff onto the stack, then pop them out … When you push things, the existing thing is gonna be hidden. Also, when you pop from the stack, and popped thing is gonna be hidden.. Now, just because they are both gonna be hidden, doesn’t mean they should lead to the same function!! The pop flow should be completely separate from the push flow.

Well, one more stupid bug, pushing more than one panel seems to stop the push animation from working >___< …

lol, indeed it was stupid… ROFL, this is just too stupid. So, while I was fixing the panel controller code, which deals with the whole navigation crap, I saw that a variable was defined like this:

private List<Panel> _panelStack;

I was like, no way! This is called a stack, it should be a stack!

private Stack<Panel> _panelStack;

Better… So, I go through the code and change Remove to Pop .. and that’s it. I leave everything else the same, including calls to the Last function. By definition, a stack puts new elements at the beginning, and I was accessing Last … Stupid. So, just changed those to Peek, and call it a day.

Come Again?

2016-07-30T22:04:18+00:00

So, after spending all that time perfecting the relationship between the game logic, graphics, and view, the game started coming together. Implementing new crap is much simpler, since everything is where it belongs, and it doesn’t take time to think about where to implement it.

With that, my trello board quickly started draining down cards, thankfully, but it’s way too early to celebrate.

Speaking of celebration, I was psyched to see the game performance go through the roof after simply turning off a clipping feature in NGUI. It had something to do with Physx processing, hogging tons of resource.. So, I wanted to check if the performance gain was real, and wanted to build the game on iOS.

As I was making changes and pushing to github, the iOS build on UCB kept failing… Weird.. It usually fails for no reason from time to time, but now it failed like 3 times in a row! Something was amiss.

Well, the biggest change I made which impacted platform specific stuff was upgrading the native dama logic plugin. I’ve made changes in the library to remove dynamic memory allocation from certain parts, in order to simplify memory management a bit … which was nice, but it’s a chore to build the library and update the server, bots, and unity with the new version.

So, after spending an hour or two updating and running the new library on the servers, after updating Pyport obviously, which is the python shim wrapping the C library, I updated unity simply by replacing the old libraries with the new ones. At that point, I hastily tested the Mac build, which worked fine, so life went on.

Now, after seeing the iOS build failures pile up, I looked deeper into the logs, and yup… the dreaded iOS native lib was causing issues because of some undefined symbols. The annoying this was the Unity itself generated C++ code through IL2Cpp that was named exactly the same name as my native code, except it had a number suffix at the end, like xyz_m12839213. This made the linker go, “Did you mean xyz_m12839213?”. I’m like, no! the suffix shouldn’t be there!

So I dig through my blog, go over git changes like 5 times to see what changed?? I did upgrade Unity to 5.4.0F2, but seriously? Even cloud build was failing, so it should be something else…

Finally, I use the best tool for inspecting libraries, nm. Using it revealed that indeed all the “glue code” generated by SWIG, to glue the C code with CSharp was missing! Going back to the Xcode project that built the native library, it seems that yeah … I was an idiot at some point and excluded that file from the build >_<.

So, finally, I mark that file as part of the built library, build the library, and off to Unity it goes … The issue now is, I’ve built too many times within a short period of time on UCB, so it will be a while till my build will be executed … Too lazy to build it locally, so just distracting myself with this blog entry and probably going out for food… Hopefully, it will be ready after that.

Once it’s really, it would be time to download it on an iPad and hope for that sweet, sweet 60 FPS glamor *_*.

The Fuck Up

2016-07-26T19:31:34+00:00

I derailed, and I derailed hard…

I was simply focused on trying to implement animation ended hooks, so whenever an animation ends, the main game loop is informed, and the next task is processed…

Somehow, during that implementation, I found a nasty part in the project, but it was working well for the most part. For some reason, I decided to rewrite it… That was such a horrible idea.

No matter what the end result is, spending 4 - 5 hours perfecting minor stuff isn’t what I should be concerned about now, and yet! In any case, I realized now that reverting might be even more painful, so I gotta get this through.

The thing I really need to think through now is how should I structure the UI code moving forward? My options are either to keep things decoupled and work through the global event system, or better yet, go back to having hard references to each part of the UI.

Even though having references couples things, but allows for much higher control and clarity, which is something I really need for this game. Hiring developers not aligned with me on this part was a huge mistake…

OK, let’s just get this “hanging” task over with, I guess…

Event Stream

2016-07-23T18:06:19+00:00

After much struggling, procrastinating, and LoLing, I have finally set my mind to it. Previously, I have explained how I should be moving the game state within the game logic, but once I started looking at the code some more, something stood out.

The graphics part was tightly coupled with the game state. I was basically reading the game state each turn, and updating the graphics accordingly. This is great for extremely simple graphics, but it’s impossible to have proper animations and sequential graphic events this way …

To give you an example, imagine how we can implement a “game turn indicator” widget. The widget is a simple arrow that points at either the white or black team, based on the turn.

Currently, I am reading the game state each turn and updating the turn! This is nice because I am not assuming the turn is simply switching colors each turn. So, if the state is reloaded for any reason, everything just works out of the box…

However, this restricts us in terms of what we can do for animations. You see, since we can’t assume the turn switch, we can’t implement a simple turn change animation, unless we hold a diff state in the GUI! Something like, if previous turn doesn’t equal next turn, then switch… This is ugly, and shouldn’t be handled by the graphics part.

Another problem I was facing with this load-from-scratch-each-time approach is overlapping events. If during an online game a move comes in, and we want to apply that, obviously an animation will be started. What if during that animation, a game ended event comes in? If we keep inspecting the whole game state each time, it would be pretty hard to accommodate all these cases…

This leaves me with the hopefully most simple and elegant solution .. An event queue. Every single change on the graphics side has to be driven by a game logic event, be it a turn change, play move, or game ended. The graphics part simply “blocks” on that queue, and process it one at a time.

This tasks seems intimidating, and I’m trying to summon every last bit of willpower I have left for this game to pull it through … I keep running into these issues, which is the most frustrating part. It’s definitely my fault for spending unreasonable time on polish before figuring these details out, but all I can do now is learn the lesson and move on.

State, Logic, and Graphics

2016-07-23T09:47:25+00:00

I am struggling to find the zen between three entities in my game:

State
Logic
Graphics

the game state is an immutable object that simply represents the state of the game. If you want to change the game state, a new copy is created. This makes game state super simple and easy to pass around with having to worry about it. As a bonus, implementing undo is literally as simple as having a stack of these game state objects.

Then we have the game logic. This is where the current game state is saved, and where the update loop happens, telling us if the game ended, or other similar events. The tricky part about this entity is that it could either be local or remote. When playing online, the logic actually completely runs on the server.

Finally, we have the graphics, which is the representation of the game logic and game state to the user. It needs the game state information in order to “know” what are the available moves, and that’s about it.

Seems simple, except when you try to organize these entities around working with singleplayer, local multiplayer, and online multiplayer. The proper abstraction needs to be in place in order for things to work properly.

The most challenging aspect of singleplayer is handling the undo action. Sure it’s as simple as a stack of GameState objects, but finding the right place in these entities to put it and update the game is tricky. Then, we have the fact that the game logic runs on the server in online games, and the game can end at any given time, if a player surrenders or disconnects, for example. We somehow need to signal that the game has ended at that point.

With all these pieces in mind … There is only one answer I can think of. Make the logic hold the current game state, and that way, we can do the following:

For the remote game logic, we simply keep the game state in sync with the server, publishing new events to the graphics part. For the local game logic, we immediately apply changes to the game state, and publish the change events back to the graphics … This should do it, I hope.

Sad part about this is, I was struggling with designing this and spent countless hours refactoring stuff back and forth because of an extremely old design mistake made in the Dama C library. In the library, I had made the GameState stateful, which was a premature optimization back in 2012!!! Haunted me to this day .. So, finally removed that optimization, cleaned stuff up, and now everything is yet simpler and easier to reason with.

All these changes I’ve been making for a whole week, btw, are for the simple reason that I need to implement “forced moves” from the server. You see, when a player fails to make a move within the alloted time, the server forced a move on them. This is very hard to accommodate without proper code design, because until now, I always assumed that moves are coming from the actual players! So, if it’s the local player, it has to come from the graphics interaction part of the game … Not the case anymore.

Smarter Game Results

2016-07-22T13:10:21+00:00

Damn, that’s why I was stuck on this all along!!

Remember the GameResult problem where I was thinking of how to represent who won? Well, the reason it was a bit more complicated than it seems is because I have “smart game results”.

I don’t generate a single GameResult and send it to all players, since there is player specific information, namely the rewards and rating delta. Instead, I generate a unique GameResult per player.

With that being said, it would be ideal to make use of this characteristic and just embed a simple enum in the GameResult:

won
lost
draw

That’s it. I tell each player whether they won, lost, or if it was a draw… Will play nicely with Elixir, and make my life easier on the frontend side as well.

Better Late Than Never

2016-07-22T13:03:01+00:00

Wow, I really should have started doing this since forever … But you you what they always say:

Better late than never

I mean, really, so many problems could have been preserved .. I’ve written a lot of them on white boards, idea paint walls, notebooks, notepads, and even voice notes, lol.

Well, time to get that task from the previous post out of the way, then it’s time for munching lunch.

Who Won?

2016-07-22T12:41:26+00:00

In Dama, a game ends under the following conditions:

A player disconnects
A player surrenders
The players agree on a draw
The game reaches a draw state
The game reaches an ended state

So many different ways to end the game, and thankfully it was super simple to streamline all this in Elixir. It’s actually quite neat, and would make for a nice post one day…

In any case, when the game ends, the backend sends a game_ended message to the frontend, with the GameResult. The game result contains the rewards the players get, if any, as well as the new rating of the players, if those changed.

For now, I need to figure out how should the frontend handle this? Since the old days, the frontend simply checked the game state to see if it’s still running, but that obviously doesn’t cover the first three termination conditions.

So, the best way to go about this is to embed the “winning” information in the GameResult payload. It would need to represent the following information:

winner -> player x
draw

in Elixir, I usually represent this using atoms and tuples as follows:

case result do
  {:winner, :white_player} -> ...
  :draw -> ...
end

That doesn’t translate well to CSharp for the frontend .. I guess my options are:

String: “winner:player”, “draw”
Enum: winner_white, winner_black, draw

I can’t think of any other viable options, really … The enum approach sounds nice, but would be painful to debug if the backend and frontend fall out of sync .. So, strings is the way to go.

A New Beginning

2016-07-22T11:53:31+00:00

to further lower the barrier of writing to my blog, I decided to start this devlog thingie. I can just do brain dumps here without even worrying about the structure, title, content, or anything like that…

Well, I actually started this because I am too lazy to create a live programming video feed. I kinda needed to make my development experience feel more interesting than just me typing stuff into a computer … So, … This is really unfocused.

In order to make my development experience more interesting, I decided to start this devlog. That’s sums it up.

Speed Tests

2016-07-11T08:29:57+00:00

Introduction

Finally, I’m back in the game (literally?), and working on the Elixir side of things for a change.

So, my Elixir backend is heavily tested using unit and integration test. For now, a total of 83 unit tests assert their way through my code, all way from checking the password hashing result, to making sure I don’t have any typos.

Life wasn’t this good just yesterday. My tests were taking 7.2 seconds to run, which for Ruby people is a dream, but for Elixir folks, we can do better.

So, I want to quickly explore the idea of tracking down the slow down and how I managed to resolve it.

Crypt Lord

So, in my case, it really wasn’t too hard to track down the slowdown. After integrating the authentication features, which uses Comeonin.Bcrypt, I noticed that a simple test is taking +300 ms to finish, sometimes +800 ms. Wat?! Why so slow?

Turns out the Crypt lord is just too heavy .. Any Warcraft fans out there?

You see, Bcrypt is slow, and that is by design. If your password database gets compromised for any reason, the hacker will have to use the purposely designed, slow algorithm to apply a rainbow table attack on the data, which would take forever. For example, just to crack a single password, it would probably take a single CPU core an excess of 5 years total.

Well, that sounds great! I mean, even though it’s slow, it is secure… However, while unit testing, we really don’t need this functionality at all. In fact, unit testing is by definition testing your code in small, isolated units. We can leave the actual Bcrypt testing for integration tests.

So, how should we go about this?

Mockery

One could easily utilize the great mock library to mock the Bcrypt module, and we are done! Unfortunately, mocking in this specific case has tons of drawbacks:

You can no longer run your tests asynchronously
You must repeat the mock setup anywhere your test invokes Comeonin.Bcrypt

This is Elixir, obviously we can do better.

DI

I am not really sure if this counts as Dependency Injection, but it sure smells like it. What we basically do is head over to our configuration files and setup the modules which provide us with certain services. Based on the run configuration, we provide a different module!

# this is in config.exs
config :myapp, :modules,
  bcrypt: Comeonin.Bcrypt

# this is in test.exs
config :myapp, :modules,
  bcrypt: MyApp.BcryptStub

Finally, head over to your main app module, and do:

defmodule MyApp do
  use Application

  ### services

  @config Application.get_env(:myapp, :modules)
  def bcrypt, do: @config[:bcrypt]

  ...
end

That’s about it. Now, instead of referencing Comeonin.Bcrypt in your code everywhere, you use MyApp.bcrypt instead. If your tests are running, the function will return the stub Bcrypt module which you have already setup to be speedy. While running in dev or prod configuration, the real Bcrypt module will take the wheel.

For the sake of completeness, this is how I setup the MyApp.BcryptStub module:

defmodule MyApp.BcryptStub do

  def hashpwsalt(password),
  do: "bcrypt-hash:" <> password

  def checkpw(password, "bcrypt-hash:" <> pass),
  do: password == pass

  def checkpw(_, _), do: false

end

As you can see, I simply replicate the functions in Comeonin.Bcrypt that I am using, and replace them with a predictable “hashing” algorithm which runs in the microsecond time order.

Conclusion

Elixir continues to kick ass, whether it was parallelism, functionality, or just pure productivity tools. With this simple trick above, the tests now run in ~1.2 seconds. incidentally, I’ve disabled async testing because I access the DB everywhere, but once I upgrade to Phoenix 1.2.0 and turn on async testing as well, tests might as well run within 800 ms or less.

Animator Trigger

2016-07-05T22:14:01+00:00

Less programming these days, and more animation design. I’ve been using Unity’s animation system to bring the game to life, and it’s a pretty solid system, I have to say.

As with all things Unity, it has gotchas, so this post will cover some of them.

Triggers

While attempting to animate an object, I ran into a bug where after setting the trigger to “state_2” it would go to “state_2”, and back again to “state_1”. It was really weird, and I had no clue why…

The first thing that comes to mind is “Exit time”. If that property is set on the animation transition, it will automatically leave the current state onto the destination state of that transition… Nope, no exit state was being set.

After fiddling around the code for a while, I found that I was using SetTrigger on “state_1” while the object was still in “state_1”. Since I didn’t allow the transition to animate to itself, the trigger would be set, and later be triggered after we leave the state.

The solution here is either to use ResetTrigger to make sure no outstanding triggers remain, or in my case, I simply allowed the animation state to transition to itself. That made the trigger clear after being set twice.

Atomic

As much as I tried, I couldn’t manipulate a property of a game object that had an animation controller manipulating the same property.. It’s annoying, but understandable, I guess…

What I mean is, if you wanted to animate the position and scale of an object, you can either tween them or use Unity’s animation system, right? Well, if you choose to use both for any reason, you can’t have both systems modify the scale or the position at the same time. You must either let Unity’s system animate the scale, and using tweening for the position, or vice versa.

Discrete

When setting up your animation states in the animation controller, you might want your animations to play discretely (separately) without any blending. Well, the thing is, Unity’s default animation settings add a certain amount of blending, depending on today’s weather.

The gotcha here is, just make sure to open the transition settings again, and set the transition duration to 0 in order to disable animation blending, and have nice discrete animations.

Obviously, that’s not always the case. For my animations, I’ve actually set the blending to full. So the animation actually happens in the transition rather than the animation state itself (if that makes any sense)

Conclusion

Unity has really come a long way.. I’ve tried it a year and two years ago, and it’s nothing like it used to be. It has finally matured to the point of actually being usable to a certain degree…

If you haven’t looked at Unity for a while, make sure to give it another shot!

Back Me Up!

2016-07-01T05:05:05+00:00

Introduction

The final post that concludes the series about game development is gonna be around the backend (server). It’s pretty big topic, but I figured I’d summarize it in one (probably) long post, so here it goes.

Back Me Up!

Wait, why do we wanna bother with a backend? Can’t we just build a game and ship it without any backend whatsoever? .. Sure you can, but you’re talking about a seriously slim chance of success, as slim as flappy bird!

Compared to most other games, flappy bird is one of the few which doesn’t rely on a backend. Angry birds, CoC, Candy Crush, .. You name it! If it has any of the features below, it needs some sort of backend support:

Achievements
Leaderboards
Matchmaking
Push notifications
User profiles
… etc

Now, if you really wanna be lean and mean, you should avoid developing a backend yourself. There are tons of services out there that provide you with basic features, which you may get away with if your game is simple enough. This list is far from exhaustive:

GameCenter
Google Play Services
PlayFab
Heroic Labs
Photon Engine

Personally, I’ve had my fair share of pain dealing with blackbox backends that leave you completely in the dark when it comes to how the server is managed. If you start growing or needing extra features .. Good luck with that! Sometimes you don’t need that, most likely (if your game is any good) you’ll want it.

I’ve gone through my share of pain and suffering with the backend, all the way from a Java based server, to a python based server on GAE, to a C++ based madness server, and finally … to Elixir-goodness.

I’ve pretty much settled with Elixir, since with relatively minimal effort, I’ve written a full-featured realtime backend. Fully tested, with unit tests and integration tests. Moreover, stress tested like mad by the authors of the phoenix framework.

Using Elixir for my game was perfect, since it finally allowed me to think about server from the clients perspective.

Once a client connects to a server, we spawn a process to track that user. When that process dies, that means the user is disconnected! SO SIMPLE!! Do you want to add a retry and wait for the user to connect back again? SURE! All your work is done within that process.

Once the user starts interacting with our backend, we send the user’s process id (pid) to workers which do stuff. For example, the matchmaker worker will take a bunch of user processes then match them against each other based on the relative rating for each player.

Besides the workers, we also have “stateful” processes, which hold a shared state between one or more user process. For example, we spawn a “room state” process, and add the user to it. Once another user process joins, we close that “room state” process, and spawn a “game state” process, add the players there, and now they can proceed playing the game!!! It’s is so eloquent I can’t help but keep admiring how superior it is compared to every other crappy (yet popular) framework out there. ehm Django. ehm Rails.

So, with that, really, it was a piece of cake to write an auth system, leaderboards, game system, “referee” worker, matchmaker, … etc. All that makes up my game city! It’s really more of a city at this point rather than a server.

With the features easily implemented, since Elixir is a productivity focused language, we also gain scalability for free. I can, at any point in time, break my backend workers into Elixir applications, push them to different servers, and have a decentralized Elixir backend, which I can easily then scale and upgrade in a super fine-grain way.

Phew! Enough with the Elixir fanboyism! Time to close off this topic and head home!

Conclusion

If you’re looking for a backend for your game, make sure you test a “vertical slice demo” on it. Trust me on this one. Don’t delay crucial features from the backend thinking that you’ll get back to it later .. That spells disaster.

If you can easily implement the “vertical slice” demo on the backend easily enough, then you’re doing it wrong indie chum. Unless you really wanna focus on the backend, like it is the core of your game, or you have some super weird requirement, just stick with a simple solution. Even a custom backend can still be simple enough to develop!

Quality of Service

2016-06-29T19:20:13+00:00

Introduction

As highlighted by the nifty list below, this is part III of the four part series about game development!

I apologize for the ambiguous, possibly inaccurate, name “game services”. I just didn’t know what to call these features, even though we use them in every single game we ever make.

The game services I am referring to are the supporting features we need around our game. JSON parsing, persistence, networking, … etc. These aren’t necessarily provided by the engine (I think it should never be), but we should easily find and integrate a third-party plug-in that provides the needed services.

So, let’s see exactly what services we need, and how our game engine can help with that.

Quality of Service

For my development needs, I needed the ability to access these basic services:

Persistence
Websocket networking
HTTP GET, POST requests
JSON parsing
Analytics
IAP (In app purchases)

Nothing fancy, really. Other games might require much more than this, including things like:

TCP/UDP networking
Compression
Encryption
… etc

Once you highlight your needed services, just take a stab at implementing a basic game that simply exercises using them. For example, if you use an engine like Cocos2d-x, you’ll find it cumbersome to integrate third party libraries for IAP, and JSON serialization is painful with C++ (especially so since they chose rapidJSON over JsonCPP).

But still, if you take a step back and realize how customizable Cocos2d-x is, you can simply nuke the services it provides for you, and integrate your own from the vast open source libraries in the C++ landscape.

That is, until you turn around and peek at what the Unity people are doing. You just wanna take a brief look, and you see them partying and having fun! You’re now like, “WTH?! Game development is hard! How come these people are celebrating at such a crucial time??”.

The thing is, Unity pretty much solved the services issue by having an Asset Store. Developers had an incentive to share their reusable services through a market place that provided them with an alternative source of income, so why not?

No matter how good an engine gets, it’ll fail horribly if it tries to be a “do-it-all”, kitchen-sink type of deal. Developers come in many flavors, and the best way to keep them happy is to allow them to bring their own tools and technologies they love, instead of forcing everything down their throat.

Conclusion

This one was brief, as it should be, and it concludes the client-side part! In the next post, we will switch over to the server, and see how game development looks from that angle.

Rev Up Your Engine

2016-06-28T02:59:55+00:00

Introduction

In the second article in this game development series overway, I would like to shed some light on the aspects that make up the game engine.

Overview:

The term “Game Engine” is arguably not well defined. For example, I can call the following code a game engine:

def run():
  while True:
    render = buffer.read_next()
    if render == 1:
      print "x"
    elif render == 2:
      print "y"

The code above has a run loop, renders some output, and can be used to make a simple game thingie .. I think.

Anyway, back to the point, a Game Engine can be graphical, with 2D and/or 3D support, or it can be text based, sound based, or any other sort of output. What I’m gonna be focusing on is a typical 2D/3D game engine, and I want to focus specifically on the graphics rendering aspects, and not other side features, like networking, persistence, memory management and whatnot.. That comes later.

Revving Up

The declaimer, once again, is that I’ll be exploring these gaming topics solely from the perspective of my game, and nothing else. So, let’s things underway.

So my game is just a checkers thingie. Moreover, I’m not planning on adding any exciting features on top of the core experience, that is playing the damn game. It would be nice if the piece would become a pacman thingie, and walk around eating other pieces, for example… That’s what we call “scope creep”.

So, I really need any engine that will give me the ability to show sprites or 3D models if possible, and allow me to animate those objects around using a jump animation. Not surprisingly, most engines fit this requirement (not my previous python game engine tho, mind you).

Another important requirement was actually the interaction. Not many game engines provide sleek and simple interaction system that handles touches and clicks in a simple manner. Unity does quite well on this end as well, so no complaints about it there.

My final, super-weird requirement, which is one of the main reasons this game is delayed as much as it is, is C/C++ support! I really needed a game engine that allowed me to plug-in my game logic and AI, which is written in C. Unity used to require a pro license for that, which pushed me away from it. With Unity 5, it’s available for all users, and they have a new plug-in inspector thingie that allows you easily create and target plug-ins for different platforms! It sure came a long way.

So, with a bit of SWIG and python magic, as highlighted here, Unity was a go!

Conclusion

Sorry for the lack of excitement here, a topic about game engines should really be much more thrilling, lol. Again, my game is super simple on the gaming side, so learn something different for a change!

Chewy GUI

2016-06-26T05:46:36+00:00

Introduction

There is always something to post about when it comes to game development it’s so vast!! So, this is a post just to talk about how vast it can be! (think meta).

Look, before you get all intimidated on me, game development can be as simple as drag and drop … Not even writing a single line of code, and still be a world-wide success. It can also be an insanely huge collaborative effort that takes years, thousands of man-hours, and cost up to half a billion dollar to make ._.

So, even though my game is obviously on the lower end of the spectrum, it’s not trivial, and has tons of features that makes it exciting to develop as well. Since this blog is concerned with software development, I’ll only cover that part.

Now, let’s break it down, shall we?

Overview:

Chewy GUI

My game falls into that group of games which are UI-heavy. For these games, a GUI system can either make or break the game. A classical example are RPGs.

Imagine an RPG game without a proper GUI system … You probably can’t, because if it had a horrible GUI system, you probably wouldn’t ever hear about it because of hard it failed.

The key here is to avoid fancy crap! The Final Fantasy GUI system is super simple, and has been reused since Final Fantasy I!! It is one of the most renowned and celebrated series of all time, and since you spend arguably most of your time on the GUI system, they must have done it right.

Since my game is heavy on GUI, I had to make sure I choose a game engine that will not fail me on that front. Some of the things you wanna look for in a game engine you’re gonna use:

WYSIWYG GUI editor
Adaptive to screen resolutions
Easy GUI navigation
Localization support
Out-of-box widgets (text fields, labels, buttons, … etc)
Simple animation support

My favorite engine until now, Cocos2d, fails horribly when it comes to GUI development, unfortunately. The lack of a usable editor, as well as the constant turmoil their GUI system goes through made me move away from it.

My eyes fell on other open-source game engines, such as LibGDX and Godot, but unfortunately, even those (at that time) weren’t usable for indie development (we are still talking about GUI side of things). I didn’t want to spend more time digging around other game engines, so I just decided to make a deal with the devil (aka Unity).

Honestly, for an engine as established as Unity, it was still very annoying to work with their new GUI system. I had issues mostly with tweening and implementing a proper navigation system… I couldn’t waste my time here, though, so I hired some else to take care of the GUI stuff.

The developers I hired were more comfortable working with NGUI than the native uGUI system shipping with Unity, which is a bit annoying for me, but they really breezed through the GUI tasks, so I didn’t have a reason to complain.

Verdict

So, Unity is the ultimate answer for GUI games? Not at all. If I were to start over, I’ll probably shock you … I’d use ReactNative or some other app framework!! My game is so heavy on GUI, it makes less sense to use a game engine and more sense to use a cross-platform app framework! It would be super easy to develop the app, and once I reach the game part, I can just use WebGL!

Well, I know, that won’t be fun at all, but I really believe it would be the best choice. Also, I think there might be other game engines that would be a better fit, possibly BuildBox, MonoGame, or some other simpler game engine (as long as it is easy to develop GUI stuff!).

Conclusion

As far as my current game goes, I’ve optimized the backend development, the service, game logic, and even the art! The only pain point left I have is a proper GUI system, and even though what I currently have is kinda good, it’s far from ideal .. Ideally, I don’t have to hire anyone just to do GUI work. Ideally, GUI would be so simple, I don’t have to write a post about it (or, at least, it can be a subtopic of the game engine).

Look forward to future posts about the rest of the topics!

Phoenix 1.2.0

2016-06-23T06:24:58+00:00

This will be a quick one, since I didn’t really upgrade to Phoenix v1.2.0, unfortunately… Why, why not?!

The main reason I was eager to upgrade was the new Presence feature. It will really streamline my code, and allow me to easily implement the monitoring stuff I mentioned in the previous post. Also, my tests are getting slow, so having the ecto async goodness would be great as well.

The reason I didn’t upgrade is because I’ve tried to, and it didn’t work out.

First off, after the upgrade, I ran my tests, and got a whole lot of long error stacktraces on my console. I’ve included a few at the bottom. The first one I tackled was the FunctionClauseError related to the handle_info stuff. After spending about an hour at the Elixir slack channel, turns out it was a bug in the db_connection library.

That’s not too bad, but seeing that this is just one error, and I now see at least three more different ones, I just can’t spend the time to dig into each and every one!

It probably makes sense at this point not to upgrade a sufficiently complex application, and wait for the release to stabilize. I have a new project in Elixir, tho, and it’s using v1.2.0 without issues. This is because I can easily avoid troublesome patterns in the new app, but it would take forever to migrate the old app.

Just a quick example of a “troublesome” pattern, that is spinning off GenServers and processes that interact with the DB. Apparently, the new Ecto sandbox stuff is nice and all, but quite restrictive. To properly use it, you either have to do it right from the get-go, or refactor your whole implementation.

That’s about it! Here are some relevant logs:

This is the log of the issue related to this pr:

03:18:08.674 [error] GenServer #PID<0.346.0> terminating
** (FunctionClauseError) no function clause matching in Socks.UserState.handle_info/2
    (socks) web/state/user_state.ex:16: Socks.UserState.handle_info({#Reference<>, {:ok, {#PID<>, #Reference<>}, Ecto.Adapters.SQL.Sandbox.Connection, {Postgrex.Protocol, %Postgrex.Protocol{}, false}}}, %{user: %Socks.User{}})
    (stdlib) gen_server.erl:615: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:681: :gen_server.handle_msg/5
    (stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3

This one is a completely random DBConnection error that doesn’t make any sense to me. I’m not sure about it, so I can’t really open an issue on Github…

Update: One of the Ecto developers told me it’s a bug already fixed on master.

03:46:11.179 [error] Postgrex.Protocol (#PID<0.218.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.270.0> exited: an exception was raised:
    ** (FunctionClauseError) no function clause matching in Ecto.Adapters.SQL.Sandbox.Pool.disconnect/4
        (ecto) lib/ecto/adapters/sql/sandbox.ex:371: Ecto.Adapters.SQL.Sandbox.Pool.disconnect({#PID<0.210.0>, #PID<0.218.0>, {#PID<0.218.0>, #Reference<0.0.4.38>}}, %DBConnection.ConnectionError{message: "owner #PID<0.269.0> exited while client #PID<0.272.0> is still running with: shutdown"}, {Postgrex.Protocol, %Postgrex.Protocol{buffer: "", connection_id: 20156, null: nil, parameters: #Reference<0.0.1.1465>, postgres: :transaction, queries: 163889, sock: {:gen_tcp, #Port<0.7617>}, timeout: 15000, transactions: :naive, types: 139307}, false}, [timeout: :infinity, repo: Socks.Repo, sandbox_pool: DBConnection.Poolboy, ownership_pool: Ecto.Adapters.SQL.Sandbox.Pool, pool_timeout: 5000, adapter: Ecto.Adapters.Postgres, username: "xxx", password: "xxx", database: "xxx", hostname: "xxx", pool: DBConnection.Ownership])
        (db_connection) lib/db_connection/ownership/proxy.ex:234: DBConnection.Ownership.Proxy.down/2
        (stdlib) gen_server.erl:615: :gen_server.try_dispatch/4
        (stdlib) gen_server.erl:681: :gen_server.handle_msg/5
        (stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3

Finally, many people have been running into this, but it seems the only way to avoid it is to properly manage your access to the DB and make sure all transactions are complete within the test time frame?

04:40:47.009 [error] GenServer #PID<0.274.0> terminating
** (stop) exited in: GenServer.call(#PID<0.267.0>, {:checkout, #Reference<0.0.2.2087>, true, 15000}, 5000)
    ** (EXIT) shutdown: "owner #PID<0.266.0> exited with: shutdown"
    (db_connection) lib/db_connection/ownership/proxy.ex:32: DBConnection.Ownership.Proxy.checkout/2
    (db_connection) lib/db_connection.ex:716: DBConnection.checkout/2
    (db_connection) lib/db_connection.ex:623: DBConnection.run/3
    (db_connection) lib/db_connection.ex:464: DBConnection.prepare_execute/4
    (ecto) lib/ecto/adapters/sql.ex:228: Ecto.Adapters.SQL.sql_call/6
    (ecto) lib/ecto/adapters/sql.ex:400: Ecto.Adapters.SQL.execute_and_cache/7
    (ecto) lib/ecto/repo/queryable.ex:127: Ecto.Repo.Queryable.execute/5
    (ecto) lib/ecto/repo/queryable.ex:40: Ecto.Repo.Queryable.all/4
    (ecto) lib/ecto/repo/queryable.ex:72: Ecto.Repo.Queryable.one!/4
    (socks) web/state/user_state.ex:20: Socks.UserState.handle_info/2
    (stdlib) gen_server.erl:615: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:681: :gen_server.handle_msg/5
    (stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3

Elixir Metrics

2016-06-22T05:42:02+00:00

Introduction

Yet another post about the Elixir game server, and this time it’s about how I should be monitoring the server metrics, such as the game rooms, matchmaking queues, and actual games…

Ideally Speaking

I guess, the ideal situation I’d like to be in is to have a time machine that would allow me to go back in time, and learn web development instead of wasting all that time with C++ and Java.

No, really. Ideally speaking, having a properly built website is awesome, but it seems like a huge investment everytime I give it a shot. You see, for a person without much frontend development knowledge, it gets intimidating which framework one should use, which template, which IDE, and so on.

Also, there is the simple reality that I am not familiar with the Phoenix JavaScript frontend library. All this meant to me that I’d be spending a full week to get something respectable out the door .. There is no way I got time for that.

Hacking a Solution

Apparently, one of the services I already have integrated, namely DataDog, has some sort of custom metric tracking. With a community contributed Elixir library, it was quite simple to get up and running with something small and simple.

With custom metrics, you can have your backend report various data points, such as counters, gouges, histograms, … and so on. The counter sounded perfect! … Except, it wasn’t perfect at all.

According to this Github issue, datadog does some sort of normalization on the data sent, and you end up with vague numbers that sort of give you a clue of the general “rate” and not “count” of the metric you are monitoring…

So, as an example, I had my :room_store process emit a statsd.increment event whenever a new room was opened, and statsd.decrement event when a room was closed … What I ended up with were fractions, since the numbers where divided over a regular interval, and are not aggregated. Doesn’t solve my issue.

MOAR Services

I’ve touched upon LogEntries in the past and how it seemed like an exciting service to try out, but never really got to that, so .. I think it’s time. Perhaps the answer lies with this service.. Maybe, just maybe, I can finish this task in a few minutes … Let’s see.

Doesn’t cut it for me, unfortunately… The minimum plan you must get to unlock real-time analytics will set you back at least $1200/yr. Maybe it’s worthwhile, but for something so uncertain as a game that’s being developed for half a decade … Moving on!

iOS

I seriously considered just wiring together an iOS app, since that’s what I am most comfortable with. The problem is, this approach doesn’t scale. After writing an iOS app, I’ll need to run it as a daemon and constantly monitor the numbers and metrics, which just doesn’t work for iOS.

Yes, Swift now runs on Linux, but that doesn’t mean it’s feasible to go and download the Swift 3 toolchain, set up all this headache, and hope there is a compatible websocket/phoenix library out there. That just doesn’t work.

Conclusion

I really don’t have a choice, my only option is as plain as day. Whatever it takes, I must implement an admin dashboard on top of my backend that will show me all these metrics. It can be the most basic, ugly looking hack every conceived, but it will get the job done and set us of the right foot!

Time for Brunch!

Firebase Service

2016-06-19T05:24:02+00:00

Introduction

It’s tons of fun running your own thing, working on your ideas and projects .. That is, until you’re broke. Now, if your thing makes enough money for you to sustain and grow, then kudos, you’ve made it. For the rest of us stuck in the chasm, we need an alternative money source.

The common knowledge where I’m from is to take a government job, slack, and work on your own thing. You see, this doesn’t cut it for me. Why would one deliberately choose to waste one’s time with stuff trivial matters? The ideal scenario is a part-time job as a programmer that also pays enough for me to sustain. Luckily, I found such job.

So, while I focus on my own projects most of the time, I then get to switch gears and work on other challenging projects. These projects give me a lot of content to write about, which is also great for this blog! So, for today’s post, we’ll be covering the aspects of writing a backend service that syncs with Firebase.

The idea is simple:

Decouple the client apps from the backend by creating a micro-service that syncs the legacy backend’s database to firebase. The client applications will then just use firebase to access the data.

Let’s first take a deeper look at these design requirements.

µService

First off, I’d like to take a closer look at the root problem that prompted the need for this task.

Root of the Problem

The main issue is that the startup has relatively high engineering work to be done, and not enough engineers to carry out the tasks. If we had a sufficient supply of engineers, we wouldn’t be bothering with this. So, to mitigate this issue, we decided it would be best to cut corners where-ever possible to reduce the engineering overhead involved.

Looking at our setup, and also due to my bias as the iOS engineer in the team, I realized the biggest drain of resources (by far) was the maintenance of two native mobile applications (iOS/Android). For a relatively simple app, I figured our first priority was to seek parity between these platforms.

iOS 💘 Android

In order to unify the development process on these platforms, we must unify the way we consume the backend API and sync data, as well as unify the GUI development process. Due to our lack of expertise in cross-platform technologies, such as NativeScript and ReactNative, we shied off from going all out with those from the start. We figured we can do some initial work to achieve partial parity first, which puts us in a better position to decide later. This is where Firebase comes in.

Firebase offers cross-platform tools to manage and deal with common services your application might need, such as datastore, crash reporting, analtyics, … etc. Utilizing Firebase would help us achieve some partial parity between the current code bases, and then, we can move to something like ReactNative when we’re ready.

Detailed Design

Currently, we have a very basic and common API that is being consumed by the client applications. After logging in and acquiring a secure token, you authenticate each API call thereafter with that token. There is one API call for now, and it is to retrieve the “user data” on a specific “date”. Something like:

/api/v1.1/user_data/YYYYMMDD/

As things are right now, the client app wakes up when a push notification is received signaling new data is available. The app then pulls the latest user data and adds it to the local store. The user also has the ability to browse old data, and another API call is issued if that data is missing.

After thinking of many ways to go about this, I had to prioritize the following constraint:

Build a micro-service completely transparent the to the current backend.

What?! Impossible! How can that be?

Well, once the user logs in on the client app, the client app simply pushes that token to the new micro-service. The service then stores that token, and continues to sync its data on daily basis. If the user decides to browse an old entry, a “forced update” request is pushed to the micro-service, which attempts to retrieve the data and write it to Firebase. At that moment, the client app will receive the Firebase data update!

With this design at hand, there is absolutely no need to change a single line in the current legacy backend! Banzai!

So, let’s write it! How hard can it be?

Well, this is actually my second time writing a Firebase service. The first time I wrote one, it was written in Elixir. It was so simple, I didn’t feel the need to blog about it … However, this time around, Firebase has changed a lot! Let’s see what these changes brought us, and what it means for our micro-service.

Shiny Fire

As we all know, Firebase has been overhauled recently with SDK v3. Google took Firebase to a whole new level (not necessarily a good thing).

The Good

I have to give Google credit for bundling push notification services, crash reporting, storage, and a bunch of other goodies with Firebase. For this job, I needed to migrate away from Parse as well. So, with these new goodies, Firebase has become an excellent alternative.

The Bad

Firebase got dragged into Google’s auth system… DAMN IT!!! This part is really frustrating because Google’s auth system is so friggin’ complicated and convoluted, you can’t make a single auth call without downloading 10 libraries, and reading 50 pages of docs.... It’s really annoying.

No matter how challenging this new Firebase system is, the task has been set. The task shall be carried out!

Carrying out the Task

I really love Python, and clinged on the possibility of using it till the end, but without avail. Without going into too much detail, let’s say I was better off writing this micro-service in NodeJs.

Indeed, with the NodeJs library Firebase has, it was super simple to get started with manipulating the Firebase datastore. By following the NodeJs getting started guide on Heroku as well, I was able to finalize the service prototype within a day’s worth of work.

It would be interesting to perhaps explore the details of this micro-service and how it works, but for now, I feel it’s quite mundane. Besides, I am already way beyond the limit for this article!

Conclusion

Writing JS code wasn’t all that bad. It was a nice change. Sure, I wasn’t able to find a good way to format string, and I had to import 10 libraries to write simple stuff, but at the end, the code was quite neat and readable. This reduces my “Languages that should 🔥” down by one. Topping that list is, of course, PHP.

See Windows

2016-06-16T05:46:36+00:00

Introduction

In this widely generic programming post, I’d like to go over the experience of building a C dynamic library (DLL) on windows. The reason I’m interested in building for windows is because my game’s logic is written in C, and my new programmer is running Unity on windows. To properly run the game on windows, he needs a version of the game logic library targeted to windows. So annoying!

Setup

To my great fortune, I had actually installed a VM manager recently, called veertu. With this app, I had a quick and easy solution to quickly run VMs running windows, ubuntu, … etc.

With that, I ran windows 10 installation, and using Github for Windows, Microsoft visual studio, Unity, and SWIG installed, the environment was all set.

Using Github app made it a breeze to quickly clone the game logic repo into the VM. After that, I quickly downloaded and ran swig to regenerate the C wrapper. I am not sure if that is required to be generated separately on each platform, but that’s what I did anyway.

Finally, it was time to figure out how to build this DLL!

Windows DLLs

After meaningless research and reading on how to build a C DLL on windows, I launched Microsoft Visual Studio and stared experimenting. The thing that worked was actually choosing “Command line application”, and selecting “DLL” on the next screen…

WHY IN THE NAME OF BILL GATES IS DLL UNDER COMMAND LINE APPLICATIONS?!

So, anyway, I chose “export symbols”, and leaved all other options unchecked, and that created the project (or solution, as it is called in MCVS).

After losing hope on figuring out how to import a folder into the solution, I simply dragged all files from windows explorer into the MCSV project. After those were added, I got a bunch of errors because of invalid includes.

Apparently, I had structured my imports cleanly using:

#include <LibName/Directory/File.h>

You see, these angle bracket header includes indicate that the compile will “know” about the location of the “LibName” directory, but in MCSV, it doesn’t. I had no idea how to point the compiler to search for header files in “LibName”, so I just renamed all the includes to be relative imports:

#include "Directory/File.h"
#include "../File.h"

That resolved the include issues, and surprisingly enough, the DLL was built successfully! That was the least painful build experience I’ve had so far, so I’m not complaining about the few hurdles above.

Enter Unity

After dropping the windows native plugin into Unity, which is running in the windows VM, I selected the plugin to play around with the plugin inspector. The plugin inspector is super, and I’m really thankful the Unity people had put it there…

I noticed that the settings for the plugin should be set to “Standalone - Windows x86_64” as well as the “Editor - Windows x86_64”. with these two options set, the game was running like a charm on the windows installation of Unity.

That was the end of it, really. I checked in the changes, and it’s time for the developer to take over from here.

Conclusion

I cannot explain how many days of this everlasting project have been drained on this freaking C game logic. To run it on the server, I had to figure out how to compile it on linux, and played around with CMake. In order to develop and test it properly on OS X, I had to prepare an Xcode project and Swift tests. Then, there is this post, where I had to run a windows VM and compile a DLL. But that’s not all…

Recently, I’ve been running game bots on my machine that automatically login to the server and play games against each other. After leaving the bots running for a few hours, I noticed my computer was getting quite slow… Yup, it was a memory leak, and the only possible culprit was the C library. The other languages in the stack (Elixir, Python) are garbage collected language, that don’t leak memory.... Also…

In order to connect the logic to the server, I had to use python and distutils to wrap the C code with a sane python script that Elixir can interact with. On the Unity game side, I needed to play around a lot with Swig and building the library for all sorts of target platforms…

QUITE A FREAKING ORDEAL!

Elixir Exits

2016-06-15T21:22:24+00:00

Introduction

While working with Elixir and building a dope server, I came across an annoying error resulting from an “exit”.

Exits in Elixir are propagated in processes similar to how errors and exceptions do in your other familiar languages. When a process exits abnormally, the “exit” hits the linked nodes, possibly crashing them if they don’t handle the exit properly.

I won’t be covering exits in details, since there is already a great resource that explains this concepts really well over here. What I will be covering is one painful place where an exit can kill your app!

Taming GenServers

While running integration tests on my server, I noticed an annoying occasional error that kept occurring from time to time:

** (EXIT from #PID<0.276.0>) exited in: GenServer.call(#PID<0.279.0>, :msg, 5000)
    ** (EXIT) no process

Hrm .. that’s a pretty obvious error. We are attempting to do a GenServer.call, but the process we are calling into is not available. Looking at my code, and indeed. Sometimes I receive a late incoming message after the “Game Process” has been terminated, causing this error to be thrown.

“Simple enough!”, I thought to myself. “I’ll just check if the process is available before I perform any calls!”

if Process.alive? pid do
  reply = Server.perform_call(pid)
end

Hmm … On second thought, that won’t do it. Care to spot the error in the code above?

Indeed, Elixir is a highly concurrent language, and if the process happens to exit after the Process.alive? check is evaluated, we will end up crashing nevertheless!! We need to improve this somehow … How about being optimistic?

try do
  # optimistically run the code expecting it to succeed
  Server.perform_call(pid)
catch
  # if it fails, we'll just absorb the error
  :exit, _ -> :ok
end

Looks good, doesn’t it? Well, kind of. The code above does work as advertised, and it may make sense in many situations, but it doesn’t make the most sense in my situation. You see, when you run into these issues in Elixir, the first solution you should think of is “how can I better refactor my code to avoid this error completely?”. Is that possible? YES!

Ideal Solution

After conceiving the previous error handling solution, I launched atom and started writing unit tests that reproduce the failing situation. After I wrote the tests, I ran then, and … they didn’t fail! WAT?!

Looking at the code, I am pretty sure I am triggering a GenServer call after the GenServer process has died .. But no exits are raised this time?

After looking closer at the code, I was actually testing against one of the methods that performed a GenServer cast rather than a call. Since performing a call is suppose to return a reply, the GenServer throws an error, since no reply is gonna be given back! But if we do a cast, there is no reply anyway, so the message fails silently…

Once I realized this neat property of call vs cast, I simply realized that the best solution is to decouple my phoenix channel process from the game state process, by always perform casts instead of calls. If there is a result to be sent back to the client, the game state will just push the event from within.

Conclusion

Even though time is super scarce these days to be writing blog posts, these types of stories and encounters are the reason I have this blog in the first place, so indeed .. It would be foolish to skip on this.

The quality of the post is admittedly bad and vague at best, but it hopefully delivers at least once useful lesson:

Always be casting!

Sysadmin

2016-06-14T05:21:00+00:00

Introduction

I’d be committing treason not showing this xkcd comic in a sysadmin related article:

With that taken care of, what is this post about exactly? Well, recently I had to wear the sysadmin hat. Here is what wiki has to say about sysadmin:

A system administrator, or sysadmin, is a person who is responsible for the upkeep, configuration, and reliable operation of computer systems; especially multi-user computers, such as servers.

Close enough.

The reason I had to do sysadmin stuff is that I needed to deploy the Elixir game server for the remote team to use. They breezed through the simple tasks and features, and are ready to integrate the networking stuff.

Traditionally, I had a lot of issues with server deployment. Issues were related to DBs, web server setup, logging, and server monitoring. Thankfully, this time around, all these issues were taken care of! Let’s see how…

Database

I decided to use Postgres as a DB, mostly because it’s the recommended DB for Phoenix. The only issue is, it’s not the most usable DB on the planet. You can get into a lot of quirks and configuration issues while setting it up yourself, so that’s why I decided to just avoid that from the get go.

A quick google search for recommended database services near my servers (hosted on DigitalOcean) showed me DatabaseLabs. The reason I felt comfortable going with them is because they only provide the thing that I needed, and nothing else. Postgres hosting on DigitalOcean. Period.

I had a lot of minor issues with the service, in the beginning. The dashboard was kinda crappy, DataGrip didn’t work, and I wasn’t able to navigate and make use of simple resources due to bad UX.

Thankfully, they had a kickass support team. I was able to communicate with them and voice my concerned, which they acknowledged and addressed fairly quickly. So, it’s definitely a :thumbs_up: from me!

Web Server

My previous server was using Apache, and it was a hellish experience to say the least. Too much crap to configure, too much intricacy involved in the way you structure your directories! There is no IDE or magic wand you can just wave here, so things didn’t go well.

Last time, I decided to use a Flask web service, and it is almost unthinkable to use the default web server shipping with Flask, hence I resorted to Apache.

This time around, I am using Elixir! Built with Erlang! Which has the superb Cowboy webserver! Yup, I decided not to bother with nginx nor Apache this time around, and just expose the Cowboy server to the outside world. It’s great, and I’m not complaining for now.

So, the way to solve the web server issue is to eliminate the need to have a web server in the first place! OK, I understand that at some point I may need to install a web server in order to do proxying and load balancing, but for now .. Wheeee!

Logging

Ah … Logging … Such an under-appreciated corner of the backend world. Up until now, working with over 5 different companies with relatively sophisticated servers didn’t have proper logging aggregation. When I would ask for logs to look at, they all pointed me to a bunch of zip files dumped on the server … Really?

The thing is, that’s what most servers come by default with. A file logger with log rotation support. This means, the server writes the logs to a file, and once a duration or files size limit is reached, the log file is compressed, moved elsewhere, and a new file is created. This is cool, but how are we suppose to go through all these logs!

Enter the logging aggregator.

With a logging aggregator, you can send all these logs to a centralized location, and this service would basically provide you with powerful search and lookup tools to properly look through all these logs.

The service I came across and found extremely useful was papertrail. It had everything I could ask for:

Regex search and filtration
Generous free trail
Elixir logger library

With papertrail, I can finally run my phoenix server in detached mode without worrying about losing any logs, since they are all sent to papertrail where I can see them update live!

A few honorable mentions include logDNA and logentries. They look super promising as well, but I am simply too satisfied with papertrail to bother with exploring alternatives.

Monitoring

We reach our final point, monitoring. During the EC2 days, I had setup New Relic, which was a terrible experience. After adding their library into my server code, it didn’t really perform well, and I could’ve sworn my server performance was degraded. In either case, I really hated the fact that I had to change my server code just to get simple monitoring.

AWS so called “Cloud Watch” is the absolute worst. I’ve also tried several solutions for monitoring any ubuntu instances, and they were all annoying to deal with … Monitoring was another thing I didn’t get right in the past.

This time around, again, there it was. The best monitoring service I could ask for, DataDog. Just be deploying a small script, I now have a daemon up an running on my server, sending all the relevant stats the the data dog server, for me to view conveniently from their dashboard.

Data dog’s great features include a fully customizable dashboard, where you choose the stats you wanna monitor, generous free tier (again), and slick UI, integration, and many other neat features. I am really happy with it.

Conclusion

I didn’t need to use Docker this time around, nor am I bothering with fancy continuous integration nor deployment… It’s time to get just the bare minimum done and ship. I’ve actually been preaching this a lot to people around me making the same grave mistake over and over again …

“We must perfect this!”, “It’s all about the technology!”, “We want the best code EVAR!!” … No.

The whole idea of startups that deliver consumer products is to MITIGATE the risk of technologies and focus on delivering the best user experience possible. Stop bogging down on the details of the technology stack, and focus on delivery, shipment and customer satisfaction. This is mostly due to endless iterations on the product facing side, not the tech side....

HEED MY WORDS!

GenServer

2016-06-08T17:42:57+00:00

Introduction

In the previous post, I didn’t have enough space nor time to cover GenServers in Elixir. This is a follow up to cover just that. Before actually talking about GenServers, let’s discuss the design pattern behind it, namely, the Actor model.

Cameras, Light, Action!

The actor model is an architectural design pattern that dictates how you structure your code. The basic building block in this design pattern is the “actor”, hence comes the name. So, what is an actor?

An actor is a self-contained entity that has it’s own model and logic. You can think of an actor as a small application running within your application! It is self-contained, so no one can access the actor’s memory directly. In order to do useful things with actors, we use message passing to facilitate data flow and perform useful tasks.

In languages such as C++ and Swift, we usually see actors running in their own threads. This is perfect for concurrent applications, since you can then make use of as much CPU cores as you have. However, threads are not light-weight and can get rather computationally expensive.

That’s why in Elixir, we have Erlang VM processes, which are super-light weight processes that provide us with the same features as OS processes, but can be created within a few µs.

Enough with the theoretical talks, and let’s talk using some real examples.

A Game Server

Since I’m building a game server with Elixir, I’ll derive the example directly from how I built the backend.

First off, I define some global actors that are always needed and provide useful, persistent functionality to the rest of the application. Some of these actors include:

The Referee

The referee is an actor that accepts a Game object for further processing. That’s it! So, for example, whenever a game ends on the server, we send the associated Game object to the referee, and move on. The referee looks at the Game object, and updates the players rating, stats, and rewards.

The Matchmaker

Another interesting actor is the Matchmaker. It has a runloop that triggers every 5 seconds to process the players in the matchmaking queue, and matches them together so they can start playing a game. You send it the player data along with the “channel pid”, and once it finds a match, it will respond to the “channel pid” with the result.

You don’t have to worry about what “channel pid” is, but for the curious, it is like the “address” of the player actor. So, we use the address to communicate the result back to the player.

GenServer

GenServer stands for Generic Server. It’s basically a server that you can spawn at runtime to server you various functionality. It is immune to race conditions and can do it’s work synchronously and asynchronously, based on the function definition (as we will see in a sec). With these features in mind, it makes it the basic building block for creating actors in Elixir.

To get comfortable with GenServers, spend some time writing various GenServers, have them talk to one another, add them to the Elixir application supervision tree, and you should be golden.

Again, for illustration purposes, let’s see an example:

defmodule MyApp.Counter do
  # enable this module as a genserver
  use GenServer

  # this is the entry point to spawn a server
  def start_link do
    # just keep in mind :ok is passed to the init function below
    GenServer.start_link(__MODULE__, :ok, [])
  end

  # here, we define the "API" of the server

  def count(pid) do
    GenServer.call(pid, :count)
  end

  def increment(pid) do
    GenServer.cast(pid, :increment)
  end

  # now, we proceed with the server implementation

  def init(:ok) do
    # 0 is the state of the server
    {:ok, 0}
  end

  def handle_call(:count, _from, state) do
    # the first "state" we pass in, is the return value
    # the second "state" is the state of the server, which remains unchanged
    {:reply, state, state}
  end

  def handle_cast(:increment, state) do
    {:noreply, state + 1}
  end
end

That looks intimidating, even to me now, since I’ve been using ExActor for a while … But, you really need to learn the basics before moving on. Just keep in mind that it get better. Once you learn GenServers the hard way, you can start utilizing ExActor to take care of the boilerplate crap.

Before breaking this down and understanding it, let’s run it!

Execution

Simply, create a counter.ex file somewhere, and paste in the code above as is. Then, navigate to that file’s directory in terminal, and launch iex. Once iex is launched, execute c("counter.ex"). This should load the MyApp.Counter module into iex.

We can then test the counter like so:

iex(1)> c "counter.ex" 
[MyApp.Counter]
iex(2)> MyApp.Counter.
count/1         increment/1     start_link/0    
iex(2)> {:ok, c} = MyApp.Counter.start_link
{:ok, #PID<0.67.0>}
iex(4)> MyApp.Counter.count c
0
iex(6)> MyApp.Counter.increment c
:ok
iex(7)> MyApp.Counter.increment c
:ok
iex(8)> MyApp.Counter.increment c
:ok
iex(9)> MyApp.Counter.count c       
3

Yay, so it works! Let’s take a close look as to why it works.

Start

To define an entry point for spawning a server, we have to define a start or start_link method. The difference is irrelevant now. The function simply calls into GenServer module to spawn and initialize a new process running our Counter server. The parameters we pass in are for initializing the server, but since this server is so simple, there is nothing to initialize.

The start function returns a tuple {:ok, pid}, which we can pattern match against in order to extract the pid, which is the server address. We need it in order to perform the operations.

Init

Init is where you’d want to initialize your server state and prepare any data or timers. Once you’re ready, you simply return {:ok, state} at the end. :ok indicates initialization was successful, and state is any arbitrary data you want the server to hold. In our case, the server’s state is just a number.

Cast vs Call

After that, we start defining the server API. What functionality do we want the server to provide? In this case, we want a way to increment the number of the server, and request that number sometime in the future.

We would like to implement the increment function asynchronously, so we don’t block the caller as the server state is updated. We do that by calling GenServer.cast in the API methods. This means, if we have a long running operation running after the call is made, the process performing the call will not block and wait for a response, it will simply move on.

However, if we want to implement retrieving the counter value, the caller process must block and wait for the response from the server with the counter value. We can instruct the caller to do that by using GenServer.call. After calling into the Counter server for the counter value, we use {:reply, state, state} to return the counter value.

The first state is just the return value to the caller. The second state means we want to maintain the current state of the server without changing anything. So, if for any reason we want to clear the counter after it has been read, we can do:

def handle_call(:count, _from, state) do
  {:reply, state, 0}
end

Finally, note the function signature of the handle_call and handle_cast methods, and how they are different. The handle_call takes one more extra parameter, which is from. from is the pid of the caller! Since it is blocking and waiting for a response, you may not have a result yet, so you want to “reply later”. This will allow you to reply whenever you want.

ExActor

After looking at the previous example, let’s look at how we can use the ExActor library to really simplify how we implement actors and GenServers in Elixir.

def MyApp.Counter do
  use ExActor

  def start_link, do: initial_state(0)
  defcall count, state: state, do: reply(state)
  defcast incr, state: state, do: new_state(state + 1)
end

I know, this is absurdly OP!! We no longer have to define API methods and handler methods. We no longer have to worry about the function signatures nor the syntax of how we reply at the end of the function. This is objectively awesome and simple, and enhances readability A LOT.

The reason you should use vanilla GenServer before jumping on ExActor is because of ambiguity that ExActor introduces. For example, in ExActor there is no pid parameter passed into count and increment functions, even though we still need to pass in the pid of the server. This is taken care of by the library, but one may be confused by such details if they didn’t try using vanilla GenServer first.

Conclusion

The journey to master GenServers doesn’t end here. I’ve missed out a lot of details because of time constraints. Details include passing in a name for the server, message handling using handle_info and other tips and tricks to make the most out of this extremely useful toy.

Don’t get intimidated, tho. Once you’ve mastered GenServer, IMHO, you are good to go to use Elixir in your own projects.

Elixir Tutorial

2016-06-05T21:30:37+00:00

Introduction

This is so nice… I’ve recently updated my sublime text theme, and the markdown plugin rendering is looking gorgeous.

Ehm, anyways. So, Jim has more ideas to share, and they still keep me up at night. I am not sure what this says about my personality, but usually I find other people’s ideas much more fascinating than mine.

So, the idea was to write Elixir tutorials (or a single tutorial, I don’t remember). I initially dismissed the idea, since I really didn’t go through any useful tutorial at all. Most of my learning was through Elixir talks with practical demos, and by reading source code on github!! But! Just sharing that experience would be a tutorial in itself, so here it goes.

Elixir?

Oh, yes. What is Elixir? Elixir is a concurrent, functional programming language built on top of Erlang. It brings you all the Erlang goodness that has been developed since 1986, with a clean and nice syntax that is actually usable and easy to work with.

Elixir excels at building highly scalable, fault-tolerant backend systems. Definitely use it to replace your background workers, Redis, RabbitMQ and other services, since Elixir is enough to do all that. As for web services, you can take a look at Phoenix, however, it wouldn’t make much sense writing a web app in Phoenix, when Node and Rails exist for that purpose.

Start Being Functional

I really believe people without object-oriented programming background are the people most suited to learning elixir. This is mainly because Elixir is a functional programing language.

Functional programming is simple and straight forward. The reason most developers I met found it difficult was because they were retrofitting object oriented concepts into this pristine language, which is a Very Bad Idea™.

So, even though I believe that, I am unfortunately burdened with an object-oriented mind while I was learning Elixir. So I have to present this post from the perspective of such persona.

OK, so before we proceed, I am presenting what really helped me get a hang of Elixir as a competent, object-oriented programmer with absolutely no prior experience with functional programming, so here it goes.

Basics

First of, you need to get your hands dirty with the absolute basics. Don’t even worry nor think about OTP, aynsc tasks, and all that exciting goodness. Just focus on the absolute basics, namely:

Pattern Matching

Pattern matching is everywhere in Elixir, and comes from Erlang. You must get this out of the way as soon as possible. While you learn pattern matching, you’ll also see the different data types in Elixir, hopefully.

For the sake of example, let’s put a few code snippets to illustrate:

# string pattern matching
string = "I am string"
"I am " <> suffix = string
# suffix is a variable containing "string"

# tuple pattern matching
tuple = {"string", 123}
{"string", num} = tuple
# num is a variable containing 123

# list pattern matching
[head|tail] = [1, 2, 3, 4, 5]
# head contains 1, and tail contains [2, 3, 4, 5]

# map pattern matching
%{"x" => x} = %{"a" => 1, "x" => 2}
# x contains 2

A few remarks on the code illustrated above:

By providing the actual value, such as "string", or "x", that value is checked against by the pattern matching engine for conformance. If a mismatch occurs on a value, an error is raised.
On the other hand, providing a variable name, such as head or suffix, this will capture the matching part into the variable for further processing.
Finally, notice that pattern matching on maps is special. Usually, when you pattern match, you have to pattern match against the whole thing. The whole list, or the whole string for example. But for maps, we are ignoring the extra keys we don’t care about, and the pattern matching succeeds.

Modules & Functions

There is no such things as classes, so we have modules instead. A module is a collection of functions, almost like a namespace, but has other uses. There is not much to cover here, really. You just define a module, write some functions within, and you’re done.

defmodule MyApp.Calculator do
  def add(x, y) do
    x + y
  end
end

# somewhere else
result = MyApp.Calculator.add(1, 2)
# result contains 3

This should be straight-forward, really. If you come from a language that uses braces or something, just replace them with do ... end.
Most languages also have return keyword, but in elixir, everything is an expression. You don’t need to worry about what that means, but just keep in mind that the last statement in the function will automatically be returned.
There are no objects, no constructors, no destructors. It’s just so simple.

Standard Library

The Elixir standard library is quite neat with most of the functionality you’d probably need already built in. You got the usual suspects, like IO, Enum, File, and such. You also have the data types, like String, List, Map, and such.

My best friend when it comes to exploring and learning the standard library is Dash. DevDocs is a free and open source alternative, but isn’t being updated as frequently. It suffers from the relatively high maintenance overhead.

Armed with those tools, you should be good to go. Don’t spend too much time goofing around with the tools, once you start building your application, you should be able to pick it up easily without impeding momentum.

GenServer

This is the last piece of the puzzle. Now, in order to actually build useful applications with Elixir, you really need to master GenServers. They are simple and easy to work with, but may be daunting to the uninitiated.

… After spending about 30 minutes writing about GenServers, I realized it’s just too big of a topic to cover here! So, fret not. I simply moved the content I write to a new draft that I will hopefully cover next. For now, spend some time on the topics already covered!!

Remarks

OK, I lied. There are some missing pieces you should learn and be wary of. Most importantly is message passing. Sending messages between processes is a super important feature to learn and understand properly if you plan on building anything useful with Elixir.

It’s quite a broad topic, but the basics should be simple enough to grasp on your own. I am out of time for now, so maybe it’s worth exploring in its own article one day.

Conclusion

The more Elixir developers out there, the better. If you ever fall in love with it, spread the word! Also, join the elixir slack channel, where the core team hangs out. Most importantly, don’t give up on Elixir except after attempting to build a full app with it. If that leaves a bad taste in your mouth, then you’re allowed to look at alternatives!

Phoenix Channel with a User

2016-06-04T00:47:29+00:00

Introduction

Phoenix is a productive web framework written in Elixir. It excels in many things, but nothing nearly as amazing as realtime websockets.

Phoenix provides an abstraction over soft-realtime communication called “Channels”. This basically handles structuring realtime communication for you. The way this structure works is out-of-scope for the moment, so please read on it from the Phoenix docs. It is really close to the Socket.IO model.

Onto the actual post!

Say My Name

So, assuming you know what Phoenix channels are by now, let’s look at an interesting problem you might easily find yourself handling: User data.

Assuming you only allow authenticated users to have access to your websocket connection (hint: good idea), you will need connecting clients to send some sort of credentials. Using these credentials, you can authenticate the user, and move on.

Since usually we would want to perform user-related actions throughout the span of the client’s connection, this means we should associate the socket connection with that particular user somehow for later reference. Easily enough, we read the docs inlined in our user_socket.ex file and find:

assign(socket, :user_id, verified_user_id)

Nice! We can assign arbitrary data to our socket, and it will be accessible on all our channels through socket.assigns! .. Not so fast. What should we actually store there? user_id? The whole User object? Something else?

User Object

For my use case, I rely heavily on the user data, like the username and rating, so I decided to just assign the whole User object to the socket and move on. This served me quite well during the development process, and didn’t cause any issues.. Until I reached the part of the implementation where I need to process and update user data.

You see, Elixir is a functional programming language, so everything is passed by value. The User object we assigned to the socket is no exception. With that in mind, it means I can’t easily update all User objects being passed around the socket and channels easily.

Moreover, I was tempted in several places to update the user data on the DB using the User object I had .. But it could be outdated! That looked really dangerous, and had to be eliminated altogether.

User Id

Let’s just follow the guide and use user_id? NO. This will result in the need to make synchronous DB calls, which will render the whole reason we are using realtime websockets useless. It would also make the code unbearable.

This could work for other use cases where the user data isn’t needed directly in the channels, but rather used in workers or background services. In that case, passing user_id around would be your best options.

Global

The first solution I tried was so stupid, lol. I created a GenServer to hold the User object, and called it UserService. This process is registered globally, and hence can be accessed from anywhere! This would eliminate the update issues as well as the performance concerns. In fact, this service would perform updates quickly on its own state, then dispatch an async DB task. Sounds great!

The issue really became glaring once I started running the tests. Unless I create a new user for every single test, I’ll have to properly clean up any running user process once each test is over… This is because after the first test runs and registers the user process globally, the second test then attempts to register the user process again, resulting in an error.

There are way to circumvent this particular issue, but that wouldn’t overcomplicate the software at hand, making future me a sad panda. Instead, I took a step back and decided to simplify the solution.

Ephemeral

Of course! First, instead of passing the process around and dealing with such an obscure data type, I decided to pass a Player object. Player is a struct with a subset of the User data, namely just the username and rating. I figured out that this is all I need, really! By moving to this new type, I am also safe from updating the actual User object using outdated info, since I can’t simply use the Player object to update the user, I need the User object.

Then! In the places where I was actually performing the user updates, I simply spawned a new UserService process! With the new process, I perform all the updates I need, then simply wait for the process to expire and perish. No global state involved, still performant as hell, and encapsulated within UserService.

Wrapping Up

Let’s quickly skim over the final architecture we reached:

Once the client connects using a token, we authenticate the user, while fetching her data. We take a subset of that data and assign it to the socket.

As the user connects to channels and starts interacting with the server, we use this subset to perform the basic operations we need. Once we need to actually update the user’s data, we spawn a UserService process, which has the available operations implemented. We perform the operations we need, and leave it to expire and die, cleaning up itself.

Conclusion

That’s it .. The fact that I can easily explain such a sophisticated topic with such simple terms is really a testament to the platform itself.

Elixir. Phoenix. Thank you.

Backend Testing Using Bots

2016-06-01T10:59:31+00:00

Introduction

With the rise of chat bots, concurrent programming, and elixir, I thought I might as well shed some light on the elixir bots server I wrote for my own needs.

The reason I wrote a bots server was mainly because I found it difficult to test my game’s online capabilities on my own. When I go the game rooms, there are no rooms, since there are no players yet! Also, not enough online events are being triggered, such as leaderboard changes and what not..

So, in order to test my game client, as well as provide a rough integration testing mechanism, I decided to write this bot server.

I Am Bot

The bot server is a simple Elixir application that has 20 bots sitting directly in the main supervision tree. Using a one_to_one strategy, the bots will recover after a crash, and life goes on.

A bot process is basically this:

Just an infinite loop that updates the GUI with the latest state and messages, and another loop that triggers the “interact” action. The really interesting part is the interaction, so let’s take a closer look at that.

Machine With Finite States

I am using the neat FSM library to create an FSM that the “bot state” uses to manage the logic. The “bot state” is just an FSM stored in the bot process, and works something like this:

So, after connecting to the server, whenever we send the FSM an “interact” event, it simply runs some logic to figure out what the next state should be. The available options are also weighted in such a way that we control the chances of the bot choosing one path over the other. This is extremely important, since for example, once the bot enters matchmaking, we give it a 5% chance to give up and leave the matchmaking queue. This is to ensure we test all possible actions!

If we were to look at some of the fsm code, it looks something like this:

### MATCHMAKING

defstate matchmaking do
  ## Interact
  defevent interact, data: data do
    action = Enum.random(List.duplicate(:wait, 24) ++ [:leave])
    case action do
      :wait ->
        GUI.update_message :gui, "Matchmaking ..."
        WS.send_heartbeat data.sock
        next_state(:matchmaking)

      :leave ->
        GUI.update_message :gui, "sick of mm waiting!"
        WS.leave data.sock, data.channel
        next_state(:connected, Map.delete(data, :channel))
    end
  end

  ## Opponent found
  defevent on_reply(event, %{"id" => id}), data: data do
    GUI.update_message :gui, "[#{event}]: #{String.slice(id, 0..7)}"
    WS.leave data.sock, data.channel
    channel = "matchmaking:" <> id
    WS.join data.sock, channel, %{}
    next_state(:mm_session, %{data | channel: channel})
  end
end

The code above represents the “matchmaking” state. So, the bot has entered a matchmaking queue, and is waiting for an opponent. While waiting, each time we call “interact”, we either continue waiting, or give up and leave. Chances are 24:1 as you can see above.

As for the defevent, this simply handles the server message if an opponent is found. The way this works is, the bot process simply forwards all incoming server messages to the bot state using the on_reply event. We use pattern matching to extract the id of the matchmaking session, then move on the mm_session state!

Closing Remarks

That’s all there is to it, really. I have to say, this server was more than worth writing. It helped my test the client app easily, as well as find bugs and issues on the server side.

For example, I run 20 bots at once and they all play and interact on the game server. After integrating the game logic on the server, I found that the game logic process hangs in certain rare cases! I would have never caught that by testing manually on a Unity client. Also, the game logic is a python process, so when it hangs, it really breaks the whole server! That’s terrible, but thankfully, resolved.

Also, I can easily do load testing by running 1000 bots or so on one machine and closely monitor the server’s performance. I am not worried about performance at all in this case, since it’s a turn-based game, but it’s easy to do, so I’d probably do it just for the heck of it.

Conclusion

Elixir is just great for writing bots, whether they are chat bots, testing sentinels, or whatever. The Actor-based model maps to bots 1-to-1, so that should really be a surprise!

Structuring Firebase Data

2016-05-31T12:17:32+00:00

Introduction

This isn’t the first Firebase post, however, it is the first since the recent revamp Google introduced to Firebase. It has a neat new logo, website, and features! It’s no longer just a datastore, but includes analytics, crash reporting, push notifications, and tons of other features.

Recently, I’ve worked with Firebase again for a client, and this time, it was about consuming data, not generating it. This presented an interesting challenge, how should the data be structured?

Basics

There are a few select rules you should bear in mind while designing your data, and they will allow you to achieve maximum efficiency while working with Firebase.

UI Driven

This is really counter-intuitive to DBAs. Usually, data is structured using ER diagrams or whatnot, and modeled in a DB regardless of how the UI is gonna present it. Not the case with Firebase.

Since Firebase is directly consumed on the client side, the data should be designed around the UI. Doing this will give you realtime super-powers and maximum flexibility in managing your apps data.

The not-so-bad downside is, you’re gonna have to replicate your datastore everytime you make a breaking change to the datastore schema, in order to not break existing clients. Not too bad.

Flat vs Nested

One simple rule:

Nest static data. Flatten dynamic data.

The real challenge here is identifying what is static data and what’s dynamic data.

Basically, if you have a nested type which once you instantiate, you really don’t want to get updates on that object anymore, that’s static data. A good example of this is the menu of an e-Commerce application. You simply create the whole menu hierarchy once, and load it. We don’t really care if a new category is added, we can reload the next time the app launches.

Dynamic data, on the other hand, is data that can change after being loaded the first time. I would go as far as saying, it’s any data that you attach a Firebase observer to. This data should be flat, for the sake of reducing the updates overhead, as well as keeping your client code clean and segregated. Let’s expand on this a little with an example.

You’ve got a user’s feed of stories, and each story has comments. So, if you decide to use a nested structure where you have:

Root/users/USER/feed/[STORY]/[COMMENT]

Anytime a comment is added to that last array, the whole object gets rewritten, all the way from the /users path, which can get insanely expensive pretty quickly. The alternative, is to denormalize your data, and just include object ids, which you later fetch using another Firebase instance:

Root/users/[USER_ID]
Root/[USER_ID]/user_data
Root/[USER_STORIES]/[STORY]
Root/[STORY_COMMENTS]/[COMMENT]

By moving everything to the root, it may seem like a chore at first, but this isolation is quite neat, as it allows you to fetch and update data in an isolated way.

The drawback this time is, what if we want to inform the user object of an update, when a new comment is added? Using nesting, the whole tree will get rewritten, and hence causing the whole tree to get an update notification.

The only reasonable way I know of is to wire things together on the client side. So, once a new comment is added, for example, you would update the UI to add that comment. As you do that, the user-related UI would be fetching the comment count from deep within the tree.

Conclusion

It’s gonna be really painful if Firebase meets the same fate as StackMob and Parse, but for rapid development, you really don’t have much of a choice but use these services…

Be Animate!

2016-05-29T11:04:13+00:00

Introduction

Apparently, the word “animate” can be an adjective to “being alive or having life”. Well, that could apply to animate and inanimate objects alike … This is getting confusing.

Well, the point is, this will be a quick post about an animation prototyping tool called “Origami”. I’ve been using it recently to test how an animation looks before the developer goes through the trouble of implementing it.

Japanese Paper Models?

Well, yeah, originally at least. That’s what the word origami initially means, however, the name was also used by facebook as the name for their animation tool, as well.

So, yeah, Facebook decided to build a free animation prototyping tool, and against all odds and expectations, the tool is actually just a bunch of “patches” on top of Quartz Composer!

Quartz Composer is an app which is part of Apple’s developer tools, and it is almost as old as time itself. I don’t recall it getting much recognition by the Apple developer communities, I guess mainly because it’s a designer tool that requires technical expertise…

So, yeah, to wrap it up, if you are a developer who excels at designing animations and looking for a tool to quickly prototype your animation ideas, you fit nicely within Origami’s target audience.

How Good Exactly?

Origami is so good, to the point that I was able to prototype two relatively complex interactions and animations within than an hour .. while on a plane (as they say in Japanese “Sasuga, Digital Nomad-san!”). Granted, I spent a few hours a few days ago playing around with the workflow and getting my hands dirty with the nuts and bolts of this tool.

In any case, the main reason Origami either shines or diminishes in a team is because of its unique workflow. Until now, every single animation system I’ve ever worked with uses keyframes (Flash, AE, Unity, Spine, … etc). That makes sense (kinda), but makes things hard and complicated once you start introducing interactions. Most people put up with it, tho, since everyone is familiar with keyframes.

Quartz Composer, however, allows you to create processing units, called “patches”. Each patch has inputs and outputs, and you basically compose your whole setup using these units. That’s it. The basic idea is so simple and elegant, you just have to fall in love with it .. Kinda reminds me of Elixir, actually, since its basically just a collection of units of work, as well. Except in Elixir, they are called “Processes”.

The final note I want to shed some light on today is maintainability. Thanks to the availability of groups and other patches that allow you to breakdown your system into smaller pieces, it really allows you to build complicated hierarchies without losing your mind.

Conclusion

I’ve tried Pixate studio, by Google, and saw other alternatives, but they were expensive (such as AE). So, I was kinda forced to try out Origami at first, but the short tutorials at their website was more than enough to convince me to test it out and ultimately use it as the primary animation tool.

Unity + JSON

2016-05-26T22:56:47+00:00

Introduction

JSON is such a trivial matter to be discussed .. but for whatever reason, it’s quite a chore on Unity.

Newton?

Newtonsoft.Json, also known as JSON.Net, is the defacto JSON serialization in C#. It’s superb, with great features, like automatic object serialization/deserialization, which reduces tons of boilerplate code.

Unfortunately, it’s not stable on Unity. It’s crazy, but it’s true. As far as I can tell, it only crashes on IL2Cpp, and does so due to some unimplemented reflection feature of some sort .. So, we can’t use this for serious Unity + iOS development!

JsonUtility?

This ships with Unity, and guarantees the least footprint + garbage, which makes it super fast and efficient for games. Unfortunately, as with most thing in Unity, it’s poorly documented making it almost useless.

Another issue is that it doesn’t have a “JSON object” class that can dynamically construct a JSON object. For example, Newtonsoft has JObject, which you can instantiate and use like a dictionary.

AssetStore?

Yeah, that’s our final hope … Tons of options are available, but since I am using a vendored library that uses the Newtonsoft namespace and JObject functionality extensively, hence I’ve decided to get a drop-in replacement.

The best option I found was this. A highly rated, well received drop-in replacement, open-sourced replacement to the broken Newtonsoft. $25 means 30 minutes of my development service, definitely worthwhile.

Conclusion

Unity is still lacking, but it has made insane leaps thanks to IL2Cpp, 2D support, UI system, and the great community. Please do consider it as an alternative to whatever else you’re using. (cough SpriteKit cough Cocos2d)

Unity on iOS

2016-05-26T19:08:35+00:00

Introduction

So, over the past three days, I spent two sleepless nights, and kept on working with one goal in mind .. Get the game working on iOS.

Wait, this is Unity! The game engine that promises painless cross-platform compatibility! Why do I have to spend over three days to make the game work properly on iOS? Especially, given the game was developed for OS X, and works great there. I mean, not too far off, you know?

Let’s see why Unity fails us here…

Enter the Walled Garden of Apples

The main issue is that Apple’s platform is “too closed”. You can’t dynamically link libraries. You can dynamically load scripts. You can’t scratch your ear without Apple’s written consent .... ARGH.

I’m not letting Unity off the hook easily, tho. They knew about this, and should work around it without exposing those issues to the end-developer. Granted, that’s (much) easier said than done.

Here are some specific details about the issues…

Native Code

The main issue I was wrestling was making my native library work on iOS. Sure, if you only wanted to compile the native functions and have them accessible from Unity, all you do is drop the *.c files into the Plugins/iOS folder. However, my library is complex, and even has C# wrappers generated with swig, so… It’s gonna be a battle.

Mono2x

First issue was related to Mono dllimport’s EntryPoint. You see, when you generate C# wrappers with swig, it also generates a C wrapper that wraps all your methods within CSharp_xxx calls (for whatever reason). So, the C# wrappers are actually functions with the original C function names, but the entry point is set to the CSharp_xxx variant.... So?

Plain and simple, this breaks when compiling for iOS with Mono 2x scripting backend. The generated Xcode project and code simply fail to locate the original function symbols, when it should be looking for the CSharp_xxx variants … This is confusing, see the example below:

// CSharp
[dllimport("MyLib", EntryPoint="CSharp_NativeMethod")]
public static extern void NativeMethod();

I hope it’s clear now .. Unity generates an Xcode project that calls NativeMethod for some reason, when it should honor the EntryPoint argument and call that name instead… That easily cost me tons of headache.

So, I tackled this issue with my trusted friend, Python. By integrating a python script after the swig generation script, I was able to transform the generated swig methods (as seen above) to the following:

// CSharp
[dllimport("MyLib")]
public static extern void CSharp_NativeMethod();
public static extern void NativeMethod() {
    CSharp_NativeMethod();
}

I basically got rid of the useless EntryPoint argument, renamed the extern method to match the swig wrapper, and finally created an alias for the other C# scripts, for compatibility. Let’s run and see … Boom! Crashes and burns.

Looking closer at the errors, seems that they are related to the callbacks swig registers to the native code. Basically, there is C# code that swig generates to pass managed (C#) methods to the native code, in order for the native code to callback the C# code in case of an exception, for example.

Yay .... it finally works! … oh, wait.

So, I ran the game on iOS, and indeed it worked for a bit, but quickly crashed because of System.Linq calls .. Apparently, Mono2x doesn’t properly implement all System.Linq functionality. Since I am in love with functional programming, I really leveraged those methods everywhere! Is all hope lost? No .. It seems if I just switch to IL2Cpp, this would resolve the issue… Let’s try that.

IL2Cpp

So, according to some resources I found online, it seems that switching the scripting backend to IL2Cpp seems to be the way Unity is moving, and is the best future proof solution. Also, it has better performance and compatibility (especially with iOS), and most importantly, it should resolve the System.Linq issues! Time to flip that switch, and hope for the best…

Boom! The native code I just fixed broke .. AGAIN. Well, at least it broke in a good way ( ͡° ͜ʖ ͡°). What does that even mean?

Well, it means the hacks I previously implemented are breaking the code! I need just remove them .. probably. So, I’ve deleted the previous python script that removes the callbacks and hacks the method names, and the result … Ye-arr… Not quite.

I got an error from the callbacks that says:

Please use “MonoPInvokeCallback”

hrm… Ok. So, back to the python script.. Let’s hack this one, shouldn’t be too hard, and it wasn’t. I simply iterated over the callback methods being passed to the native code, and added the following decorator:

[MonoPInvokeCallback(typeof(DELEGATE))]

Instead of DELEGATE, I have the actual delegate definition that matches the callback signature … Let’s run it, one more time… Success! IL2Cpp banzai! It was unfortunate that I wasted time on the previous fix, but wasn’t too bad.

Being a Perfectionist

Because I unfortunately always strive for perfection, I was like:

What if I can build the Swig C# code into mono DLLs, then conditionally compile them to each platform using the plugin inspector?

After much time ~~wasted~~ invested into learning how to build and package mono code on OS X, it just failed horribly. Managed libraries can’t contain unmanaged code calls, it seems, so that had to be thrown out the window.

So, I went back to my old friend, again, and asked him to do me a solid. While processing the PInvoke C# file with all the decorators and such, I add some more python code to do the following:

callback_decorator_template = """{indentation}#if UNITY_IOS
{indentation}{ios_decorator}
{indentation}#else
{indentation}{original_decrorator}
{indentation}#endif
"""

Basically, another regex processor that replaces the default native calls decorators and adds this conditional compilation directive based on the platform. This way, we can maintain iOS and OS X support!

Conclusion

Yeah, quite a hearty post, with lots of issues and stuff .. Even though, I’ve yet to cover all the details! For example, the Xcode build process, the swig integration, the mcs tool to build the mono libraries, Unity’s plugin inspector, … and so on.

Well, at least the general idea is out there, hopefully the small details are easy to figure out! (yeah, that’s rarely the case).

A Game Dev Tale

2016-05-23T22:44:06+00:00

Introduction

The main reason I sit on the computer these days and write code is mostly to ship a game .. A game that has become a serious stigma I can no longer ignore.

There are so many tales and angles to this journey, I cannot hope to cover it all in one post.. In fact, I started this post to write about integrating C with Elixir, and the introduction ended up being so long, to the point that it warranted a post in itself! So, let’s leave the C and Elixir part, and take a look at how the development the game’s backend underwent.

So.. Come a bit closer and hear, for there are some lessons for the programmer folk.

Seeking the Truth

So, during my insane 4-year game development journey, one of my main sources of distress was the Game Logic. Since the beginning, I’ve decided to use C for performance and portability. I really need C performance, and maybe even parallelism if this game makes it. Then there is portability, which is for the sake of running the code on the client and server… Here is where things get hairy.

The Fox

The first server I wanted to use was SmartFoxServer or Photon server (mid 2013), but they had “lacking” capabilities, or so I thought. SmartFox didn’t have great C++ client library support, and Photon didn’t allow for custom game logic to run in the cloud (urgh, I didn’t even need that back then).

When I say the client library was bad, I mean it took me around a week to just compile it. It wasn’t just me, people all over the SmartFoxServer forums were confused with all the issues the library had. Once it compiled, yeah, it crashed, and was no longer a feasible approach..

That made me go all custom backend, and start writing a game server using GAE and python.

The Snake

The Python road was kinda documented on this blog a bit (late 2013 - mid 2014), and it was an utter disaster. For whatever reason, I thought I was doing things right, by incrementally building features, without a clear overview nor goal of the final backend…

To give you a glimpse of the issues, we can separate the issues into different groups, based on the source of the issue:

GAE

GAE was a promising platform back in 2013, but severely lacking. Getting SSL calls to work locally, you had to hack the GAE code. To run sockets, you had to add a payment method, which they promise they won’t deduct from, but they eventually do. And of course, to run background jobs on GAE, you could only schedule 1-minute interval cron jobs … How limiting.

Testing GAE was horrible. Period. You gotta setup a testbed, do some various crap, just to be able to test a limited part of the code … Damn… I ended up writing functional tests rather than unit tests, which as we all know, aren’t viable for quickly catching bugs during development.

An important problem that I completely ignored here was the game logic. I had no way to run C code on GAE, so I was gonna depend on the client code being 100%, which will never go well .. Yeah …

Now, after gaining some experience about backends, I could’ve solved the background jobs as well as the C library stuff using RPC. GAE has great RPC support, but oh well.

REST

REST APIs are horrible for games. I don’t care about all the hacky workarounds that people have built around REST to make it work for games, it is just horrible… Every small feature I wanted to implement on top of REST took tons of hours of design and implementation. Think about how to solve game rooms, matchmaking and game sessions using REST apis, it’s really awkward.

The best way I learned to make REST work was to wrap everything in timers. So, the backend has an “inactivity timer” to determine if a client is still connected. The client has a “heartbeat” timer to poll the backend for results, as well as inform the backend that it is alive .. Yuck …

Webapp2

The default web service framework shipping with GAE back then, and to put it simply, it ain’t Django. It was severely lacking, resources were scarce, and for a frontend developer like me jumping to backend development through this .. It was just a bad idea.

Scalability

That sounds really appealing to most developers, but is just a another painful obstacle between you and shipping your crap. The seamless scalability GAE offers is at the expense of various restrictions, really hurtful at time, with of course, the added overhead of learning new design patterns to do simple stuff.

Thanks to scalability, the recommended approach I came across was to use secure cookies. This makes your backend stateless, and the client maintains the state with each request .. Countless headaches were there. Then comes the shared backend state, which is just inevitable in games. You had to use the memcache thingie, and writing models and using it was also yet another pain.

All that to say, it wasn’t a good idea to go head first into a new technology like GAE back then, without clearly knowing what I was after.. Yeah, sounds obvious, but … oh well.

Moving On

Yeah, that fell flat on it’s face… So, let’s write C++!

The Goliath

So, at some point, I decided to go full-fledged custom C++ (mid 2014 - mid 2015). At the same time, I looked at the UI designs I’ve had made about a year ago, and decided they are no good. I was gonna work my ass off on this C++ code base, the game better be worthwhile! So, it was time to overhaul the whole thing, design and development.

As for the code part, so much time was spent designing and implementing C++ code, on both the client and backend. The main reason I considered C++ was my discovery of the Poco libraries.

Poco is a really neat set of open-source C++ libraries. Browsing their code, I had faith that people can write sane C++ code, without losing their mind. So, following their lead, and using most of their libraries, I was able to write a “working” game frontend and backend in like .. 6 months.

Now, “working” doesn’t mean complete .. Actually, it was extremely far from complete. Even though it wasn’t a horrible experience, it was ideal for a one-person team. Juggling the frontend and backend code bases in C++ was extremely painful, I needed help, and I needed it fast.

So, during my stay in SF, late 2015, I interviewed a seasoned C++ developer, actually a core Poco contributor. He was great, and I after setting everything up, preparing the tasks, and putting together a plan … It struck me.

I was in a hostile-type place, with 2 other developers.. One of them recently graduated from a code bootcamp, and he and his mates already had a working platform with some pretty neat functionality. That really dragged me back into reality… Why not? What’s so special about them? They used node! In conjunction with Socket.IO! … URGH!

I quickly scrambled to see how easy it was, and I had a working, realtime server within the hour, just how cool is that? .. Yeah, I had to call that C++ developer and tell him not to show up for work (remotely). I really feel bad for backing off like that, but it had to be done .. I needed a sane technology like node, and to end this C++ madness.

The Elixir of Immortality

Yeah, by late 2015, it was time for me to just take a deep breath, sit back and really weigh out my options here ..

Node was an extremely viable option, but I simply hate JS (pre-ES6). I desperately looking around the web for solutions.. People seem to like Ruby, so I gave that a go. Nope, won’t work for me. It had so much of the crap I hated about python and GAE, in the sense that you do a lot of documentation reading and configuring rather than writing your business logic. At that point, it was gonna be Node…

God bless this man, for he has planted the Elixir seed in my head. He is a Ruby developer by trade, but was fed up with Ruby, and wanted to try something new. He mentioned Elixir in the past, and it clicked with me while I was there .. Yeah, why not give that a try?

Thanks to all the amazing content online that talk about Elixir, and using it for game development even, I was able to come to peace that .. Yes. This is the technology most befitting for this project.

What Now?

No, I haven’t shipped yet, lol. However, For the first time in the history of this game, the backend is way ahead of the frontend. In fact, there was only one way I realized I can ever ship, and that is by using Unity and hiring a developer to do everything, lol.

Unity is great, it’s a kitchen sink that just gives you everything you need to ship a great game, with the expense of forcing you to do things its way. I was able to get some good results on my own in Unity, but nothing close to what a veteran was able to do within a single day.

Yeah, today marked the first day working with a freelancing studio, and their progress was nothing short of staggering. Looking at their pipeline, I could see reused scripts, custom Unity editor controls, and tons of other important details they built during their 5-year experience with Unity. They went all in, and it is definitely paying off.

What now? Time to trust these guys with the Unity side, and do my best to support them where I can while implementing final touches on the backend! If this work out, that would mean the first 3.5 years where nothing but a learning journey, while the final 6 months were the actual work that made it into the game .. Since the backend, frontend, and design were all remade! (except the C game logic, lol. That is the only piece of legacy that survived this harsh journey).

Conclusion

I am not even trying to obscure the root cause of this delay in the game development process. The root cause for the game taking 4 years up till now is most definitely due to my lack of experience and focus. All the technologies I mentioned work great for their purposes, but when you decide to go with something, don’t hear the preaches.. Just cast it on your requirements.

See Sharp

2016-05-22T18:26:04+00:00

Introduction

In order to see sharp, we eat carrots! Unfortunately tho, that’s now how we C#. C# is a language brought to us by Microsoft in an attempt to beat Java (I think?). Since the dark ages, I misconceived C# as a Java close that didn’t really bring anything new to the table … I’m severely mistaken.

I guess the main reason I am writing this is because our old friend Jim kept telling me how he wrote setters and getters in C# the “Java way”, by defining the set & get methods for private properties. Even tho that isn’t a bad practice, he was distressed about how he missed the getter and setter syntax C# has. All you need to do is read more C# code! Or, read this post.

So yeah, I guess this is a good C# crash course for those of us jumping head first into Unity … Everytime I remember Unity uses C#, and an old version even, it makes me just so frustrated and want to rant for hours about how unfortunate that decision is. Let’s just forget that for now, and focus on C#.

Programming Carrots

Getters and Setters

Well, let’s get this one out of the way already. C# allows developers to easily override a property’s getter and setter behavior using a very concise syntax that doesn’t involve defining new methods:

class Test {

    public string firstName { get; private set; }
    public string lastName { get; private set; }

    public string fullName {
        get {
            return firstName + " " + lastName;
        }
        set {
            var components = value.split(" ")
            firstName = components[0];
            lastName = components[1];
        }
    }
}

First off, please don’t use this code. A space is by no means a good delimiter to use for splitting the first name and last name.

Now, you can probably tell that the firstName and lastName properties, even though declared public, their setters are declared private. So, they can only be changed within the Test class. This is a very cool way to quickly declare readonly properties.

As for the fullName property, it is a computed property. This means, everytime you try to access this property, you trigger the code within the get block. Also, whenever you assign a new value to this property, the set block is triggered, and that new value is available in the value variable, which is implicitly defined by the language.

Lambdas

Java didn’t have lambdas until very recently, but C# had those for a long time. Moreover, they have a really clean syntax, me likey. There is a downside, tho, and it is how the lambdas infer their types. You should either construct the lambda by explicitly defining the type (you can’t use var) or by passing it directly to an argument expecting a callback.

Here is where things get hairy. To define a lambda type in a clean way, we usually use typedef in C/C++/ObjC. C# doesn’t have that, but instead, it has delegate. Such an unfortunate keyword, but oh well. Let’s see how that works:

// this won't work, since we don't know the type of x
var lambda = x => return x + 1;
// first declare the "function type"
delegate int CustomCallback(int x);
// now, we can define a lambda according to the signature above
CustomCallback lambda = x => return x + 1;

Just a quick tip, here are different ways to declare a lambda:

// pass a block of code, must terminate statements with ";"
CustomCallback lambda = (x) => {
    var y = x + 1;
    return y * 2;
};

// simple and concise
// no need for ";" at the end
// no need for a return call
CustomCallback lambda = x => x + 1;
CustomCallback lambda = (int x) => x + 1;

Events

This one is a bit weird, but definitely unique to C#. Events. Another unfortunate keyword reservation event.

Events basically help you simplify the observer pattern implementation, or even the reactor pattern. Instead of writing a data structure to hold the observers, then adding custom methods to add and remove observers, and finally iterating over the observers to trigger an event, just use event.

class GlobalEvents {
    // this is the delegate declaration we saw earlier
    // the event handler will be called for every new X value
    public delegate void EventHandler(int newX);
    public static event EventHandler OnXUpdate;
}

That’s it! We first defined the signature of the callback handlers we expect, and then declared a static event variable which we can register to. Let’s now use it:

// add a handler that will print X
// we define so we can later remove it
GlobalEvents.EventHandler handler = (x) => Console.WriteLine("x: " + x);
GlobalEvents.OnXUpdate += handler;

// somewhere, when x gets updated and we want to send an event
if (GlobalEvents.OnXUpdate != null) {
    GlobalEvents.OnXUpdate(newX);
}

// when we decide to remove it
GlobalEvents.OnXUpdate -= handler;

So, in this case, we simply registered a lambda function to print the value of x using the += operator. By adding the handler, OnXUpdate event is no longer null, and actually has handlers. So, when we later decide to invoke it, it will trigger all the callbacks with the newX value. And finally, when we are done, we use the -= operator to do the cleanup.

Constructors

This one is a quick win. Instead of bothering with repeatedly initializing an object by setting various properties, you can simply initialize the parameters using some sort of initialization block:

// mendokusai
var a = new A();
a.x = 1;
a.y = 2;

// yatta!
var a = new A() {
    x = 1,
    y = 2
};

More?

I am missing some important C# features, especially string interpolation using the $, but that’s just because Unity doesn’t support it :( For the curious, I mean this is valid C# code (outside unity):

var obj = new Object();
Console.WriteLine($"object: {obj.ToString()}");

Conclusion

Onto the OSS we march! Please don’t use the Unity forums nor packages as a source for learning C#, because they are (unfortunately) really poorly written. Most packages, even the most renowned animation libraries, have absolutely horrible code and practices (sorry to say, but it’s true). Instead, divert your eyes to Github, and tho shall see the light.

Alchemy

2016-05-22T01:29:27+00:00

Introduction

At least, that’s what I feel like when writing Elixir code .. Like an Alchemist… Constructing a bunch of flasks and burners, interconnected with a series of glass tubes, working seamlessly together to produce meth- erm.. I mean, Magic!

You High, Bro?

I can’t deny the high feeling I get when working on Elixir. It’s just such a foreign language when compared to the crappy language I learned programming through (C++!). Without the static typing, without the generics, without the classes, without the inheritance … Functional programming at it’s best, coupled with process-oriented architecture. Let me just spit out a few nice Elixir encounters I’ve had.

.. On second thought, you should probably get an idea what the project itself is! Well, it’s a game backend! Yup, a game backend, with user profiles, matchmaking, rooms, leaderboards, game sessions, .. The whole 9-yards, basically. For a one developer, with absolutely no prior experience in shipping Elixir nor a game backend to do that would be considered insanity .. But here I am, 5 months into this project, with the most stable backend code I’ve ever written.

DOOOOSHITE?!

It’s Erlang

Built on top of Erlang, this robust, mature, and fully-featured beast, Elixir is usually the only technology you’ll need to run the show. Within the Erlang ecosystem, you can take care of logging, background jobs, routing, … etc. This cuts down the complexity tremendously!

It’s Functional

Functional programming is by definition simple. The fact that no data is mutable just takes one more thing off your head, “Will an object change under my feet?” .. NEVER.

It’s Concurrent

The fact that it’s concurrent causes lots of misconception that such a feature results in complexity and overhead … NO! That can’t be any further from the truth.

The fact that it’s concurrent means you are safe from race conditions. It means you can isolate your code with minimal effort, and safely contain any error from causing the whole system to go down. It means .. profit.

Am I Truthly?

Ok, how can I convince anyone with all that rant above without some stories, eh? Sore-wa, dame desu.

Player State

First, let’s take a quick look at a very quirky problem that usually takes a ton of work to sort out .. Player state.

You have player states everywhere! A game session contains the game state along with the connected players. A game room contains the players that joined, and could be chatting with one another. A matchmaking state tracks if the players accept the challenge or decline … and so on.

I am not even interested in exploring the solution in a traditional OOP language, since it would involve some sort of central crappy thing, OS threads, and stuff like that which just makes the system ever so error prone.

What about Elixir?

It’s literally as simple as spawning an Elixir process (Actor) that would manage the session. That’s it! The process would also monitor the player state, and make sure if the user disconnects, it processes the state accordingly. Let’s take an example.

A player opens a game room by sending a “join” request with a certain ID format. Once the backend detects that the player is trying to join with an ID and room meta data, it will spawn a new “room state” process, assign it the room meta data, register the process globally using the room id, then add the host player.

Once the room is added, we broadcast its availability to all players. Once another player decides to join, the player sends the same “join” request, with that room id (but no room meta this time). Once the backend processes that request, it can easily fetch the room process using the ID provided, since remember! We did register the process globally using the room id! So, fetch it, add the user, and you are done!

Well, at that point, the room state process will notice that it has 2 players, which means the room is filled! It will signal both players to proceed to the game, and broadcast an updated to the other players that the room is no longer available … la fin!

Conclusion

I can never cover my journey with Elixir in one post, nor ten, but hopefully this will give you some urge to unravel this mighty beast, and look beyond the primitive tools education casts on us!

Reboot!

2016-05-22T00:59:54+00:00

Introduction

Onto the reboot! Reboot what you ask?! Reboot the very fundamental reason I blog for! Looking back at how long it has been since I blogged, I realized that my actual productivity has an inverse relationship with the amount of articles I dish out .. So, this leads to two choices:

Kill the blog (NO WAY!)
Approach blogging differently

It’s not You; It’s Me

As I continued to write blog posts, I slowly felt inclined to write the highest quality posts I can muster, with details, graphics, references, … etc. None of that matters!

Looking back at the oldest posts, they were just purely spontaneous brain dumps that helped me track my progress, as well as have some sort of reference in the future regarding my work … So, that’s what it’s gonna be!

So, onto a reboot to clear these undesirable mutations, and restore the state to it’s initial pristine condition! … Yes, I am so excited about blogging again, my brain is just conjuring all sorts of phrases and words I had no idea I ever knew …

Conclusion

Check out the next post, coming out in a few…

Swig Swag

2016-02-16T11:31:02+00:00

Introduction

Whenever programmers talk about programming languages, and how amazing those new languages are, they always end that discussion with how kick-ass C still is! It’s amazing how this primitive language is still at large and being used throughout the industry.

While working on my game, I have built the core artificial intelligence in C for maximum performance, and boy let me tell you … That increased the performance by orders of magnitude.

Performance wasn’t the only benefit I got from using C. Believe it or not, portability was another great feature, and the fact that I am using Unity to develop my game now didn’t change the fact that the AI is still written in C. Using C# and C in Unity has to be credited to SWIG, the star of our show tonight, so let’s see what that’s all about.

SWIG: Something Wonderful In Gloves

SWIG is actually a very simple tool as a concept, but the value it offers is just too great. The basic idea is, integrating your C/C++ code with (almost) any other language.

You see, most languages support calling C libraries natively, because C is the king that they turn back to when raw power is requried. Python has ctypes, C# has Marshaling, Java has JNI, ObjC just works with C, and so on… So, where does SWIG fit in that picture?

SWIG tells you as a developer, define the interface of your C/C++ library in a *.i file, and I’ll do the rest. Whoa! What about all that horrible JNI code I had to write? What about those weird marshaling rules C# requires? Nil of that is required… Sweet!

SWIG holds true to that (for C at least). I have simplified the interface of my AI a bit, and defined a very simple *.i file:

%module KodLogic
%{
 /* Includes the header in the wrapper code */
 #include "DamaCommon.h"
 #include "DamaBot.h"
 #include "DamaGameLogic.h"
 #include "DamaUndoStack.h"
%}

/* Parse the header file to generate wrappers */
%include "stdint.i"
%include "stdbool.i"
%include "DamaCommon.h"
%include "DamaBot.h"
%include "DamaGameLogic.h"
%include "DamaUndoStack.h"

Yup, that was it. All I had to do then was run SWIG, and tell it that I want to be able to use this library from CSharp, and it replied, “Your command is my output”.

There isn’t really much else to cover. SWIG gave me all the CSharp files I needed to use the library, and I simply linked the library and dropped the CSharp files into my project. Everything was fly!

Conclusion

I guess if you ever had doubts about what game engine you’ll use, or what language you’ll be developing on as you write some core functionality, just keep in mind that C is there for you. Sure, it may not have Generics, dynamic collections, and other basic features, but it wasn’t so bad writing all those linked lists in pure C! lol, I’ll probably switch them later to a better dynamic array implementation, but for now, this will do.

MonoDevelop For N00bs

2016-02-16T09:53:31+00:00

Introduction

Over the last weekend, I had the pleasure of committing to learning all about MonoDevelop. For those of you not in the know, MonoDevelop is the default IDE shipped with Unity.

In this article, I wanted to share my experience, and explain why I hated MonoDevelop so much in the past, and why I completely changed my mind now.

Setting Up

For whatever reason, it doesn’t matter, I haven’t had success in the past writing simple mono libraries using MonoDevelop. It was probably due to the fact that MonoDevelop refers to “projects” as “solutions”. That confused me quite a bit in the past. Also, when linking with libraries, you had to refer to them as “references”. Above all that, you had two different “solution” settings, the parent and the actual solution… Talk about being spoiled by Xcode!

So, anyways, the take away here is, you can do all the usual tasks any IDE does (duh!), but you need to be careful finding out what they are called, and how they are accessed.

Linking

So, now we know the basics of MonoDevelop… Or at least, the basic terminologies. How about doing something useful this time? Let’s try linking some references!

First, thing you might try is drag and drop. That will fail horribly, as MonoDevelop uses a very unique way for linking references. All the referencing is actually done from within the references folder, located under your MonoDevelop solution:

Linking System References

So, in order to link a System reference, look no further than the first tab. Over there, you can simply choose your pick of System libraries to leverage withing your solution. This is actually the same way you would link to an existing solution within the same MonoDevelop Workspace. That solution will simply show up on the first tab, and will be built accordingly as the main target gets built. Nifty!

Linking To External Packages

It took me forever to figure this one out … I kept dragging and dropping external libraries, adding them to /usr/local/lib, and even trying to link them through terminal. Non of that worked. I was getting frustrated when the solution was staring me in the face. It was under the third tab all along. Just navigate to that tab and hit “Browse…”, import the library, and link!

Testing

I had a real issue developing on MonoDevelop because I was never able to write proper unit tests. I found “NUnit”, but seriously, there is absolutely NO resource out there I found that explains how to integrate NUnit with your MonoDevelop solution. Most articles assumed you had NUnit integrated for some reason, maybe because most C# developers are on Windows? Who knows … But the way to do it is to download the library and add it as a reference using the explanation above as a reference.

Once you have the NUnit reference setup, you are all set! Simply follow any NUnit tutorial on how to write tests, and those are dime a dozen. Once you do, you will be rewarded with a really amazing UI that the MonoDevelop team have built for unit test writers:

Conclusion

For those OSX users develop Unity apps, I am sure you feel my pain as I write this… I actually spent a whole day porting my logic code from C# to pure C, just because I could use Xcode Unit Tests!! After doing that, and integrating the C library with Unity using Swig, I realized that I can’t keep writing everything in C… It’s just too much of a commitment. That’s when I decided to just spend a whole day learning this MonoDevelop tool and getting a basic Unit Tested library up and running… Well, it didn’t stop there, actually. I was also able to get a basic C# client talking with a Phoenix server over websockets, yaaay.

Contributing on Github

2015-12-28T16:40:38+00:00

Well, here is a video blog post about contributing on Github:

Youtube link

Kitz is Live!

2015-12-19T20:11:44+00:00

Kitz Debut

I am officially announcing the simultaneous release of four Kitz. Well, Kitz is the name of the project, and it is a collection of quality Swift libraries. Four of those libraries have been released today! In celebration of this prosperous release, I propose a toast! (Make sure you come back and not enter the youtube paradox)

These libraries were suppose to be a quick release of components that I wrote for an app, however, nothing is ever quick. It became apparent that the libraries had a very rigid API that fit that app, and that app alone. I abandoned the idea for a while since…

It came back to me after starting the next app, and needing those libraries badly. Now, with two different apps that need to use these libraries, I was able to generalize the APIs and make the libraries a lot more flexible .. That took time, along with all the tests and documentation. About 1.5 months worth of time.

Thanks to encouraging contributions and feedback from various developers, I started to see potential in these libraries if done well. I guess I tried to invest just enough effort as to not make it suck, while still not putting in too much effort as to not regret it if it fails..

The Goodies

I’ll go over the most important points that makes Kitz super awesome.

Self-Contained

Each library is self-contained, with minimal dependencies. The only dependencies allowed are Apple frameworks, since let’s face it, they are still needed instead of reinventing everything.

Minimalistic

Each library is also so small and tries to do one thing, and one thing only, right. You will notice that the libraries don’t need extensive documentation. The constraint here is to actually limit the documentation to a single page, the README file.

Real-World Requirements

All these libraries came from real-world apps pushed to production. They are not contrived nor built for the sake of using every single feature in Swift. They are genuinely built to make the life of the everyday Swift developer easier.

Testing

Every library is full tested, in production, as well as through the extensive unit tests. Unit tests are an extremely important part of these libraries, since we would like to evolve them as fast as possible without worrying too much about regression.

Continuous Integration

Last, but not least, continuous integration, brought to you by Travis. This step is crucial to embrace the opensource community and their contribution. Without CI, it would be a tedious chore to manage contributions.

Conclusion

Give these libraries a try, open issues, submit PRs, I’ll even add you to the organization if you like. It’s meant to be a collaborative effort in a fun environment. Also, star, share and spread the ❤️.

Nomad Review - San Diego

2015-12-16T23:27:10+00:00

Introduction

Since this has turned into a “Digial Nomad Journal”, it is only appropriate to mention the places I’ve been to and how that relates to the work environment. As I write this, I am one stop away from the final destination, but there is a lot to cover, anyway.

San Diego

This was my first destination in the US. A friend invited me over at a time when I needed to move to somewhere new. Arriving during mid September meant I could enjoy the best weather during the whole year. The first month there was perfect, considering it was new, lots to explore, and the weather of course.

SD has a great location, being near LA and Irvine. This city lies in south California, just above the Mexican border. You must have access to a car over there, since it’s a very sparse city. With car access, you can also consider a 3 hour drive to LA for attending special events or the likes.

Once I got over the SD hype, I realized there isn’t much to it beside that. It was hard to find working places. The downtown was nice, but not extensive. Almost everything was over-priced, but that’s all of California, really. I do have to say, they had some great food options. Sushi, Gourmets, fast food, and Indian options were plentiful.

One really painful realization was the lack of a real tech scene or tech presence. Being software developers we usually thrive in communities and collaborative environments. That was non-existent.

To wrap up here, I’d give this place a stay of 1 - 2 months max. If possible, try staying near the downtown, since that’s where all the stuff is happening. It is a very fun place to be for tourists, no one can deny that, but not so much for digital nomads.

Conclusion

With San Diego out of the picture, it was time to aim for something new. The next destination was Boston, MA. We will hopefully be covering that in the next part.

Constructing URLs in Foundation

2015-12-14T19:23:26+00:00

Introduction

While building Appz, we had to deal with a lot of URLs; constructing URLs for both opening other apps as well as websites.

Once the first Kitz contributor started submitting pull requests, it was apparent that it was error prone for developers to add support for other apps. The API was expecting a fully constructed URL string, so it was the responsibility of the app URL definitions to perform the escaping and full URL construction, which didn’t make sense. That is too much burden for all the App definitions collectively.

Instead .... Actually, it’s about time we transitioned into the blog post!

EZ URL Definitions

So, the API was rewritten to make the definitions easier to write and maintain. For that, a Path class was introduced. It simply gives a very convenient way to define a url path by assigning a list of path components as well as query parameters, without worrying how they will be encoded in the end.

struct Path {

    var pathComponents = [String]()
    var queryParameters = [String:String]()
}

Honestly, the Path class is a bit redundant to something that NSURL or NSURLComponents should already do, but they require lots of boilerplate code to get right, so we opted-in for a custom class.

EZ URL Encoding

So, now we have the path definition, but how do we actually encode it to a URL? I stumbled upon a bunch of answers on stackoverflow, but almost all of them had annoying drawbacks … Until I came across NSURLQueryItem.

First, let’s take a look at the classes Foundation offers for dealing with URLs:

NSURL: Base class for representing a URL
NSURLComponents: Factory class for assembling URLs (iOS 7+)
NSURLQueryItem: a name/value pair for a URL query (iOS 8+)

NSURL

I’d assume anyone who spent a few weeks writing Foundation based application will have definitely come across NSURL. It is used in the NSFileManager and NSBundle APIs for dealing with filepaths, NSURLSession for web resources, and many other use cases.

The biggest caveat about NSURL that developers miss is its immutability. It has a very shallow API for constructing and changing the underlying URL, and when it does, it would return a new NSURL instance. The most famous method that fits this criteria is URLByAppendingPathComponent:, which returns a new NSURL? after attempting to append the passed value as a path. Not very helpful, as we well see later.

NSURLComponents

As mentioned, this class acts as a factory for NSURL instances. You can initialize an instance of NSURLComponents, then start customizing how the resulting NSURL instances will be. Let’s see an example:

let comps = NSURLComponents()
comps.scheme = "https"
comps.host = "github.com"
comps.path = "/SwiftKitz/Appz"
comps.queryItems = [NSURLQueryItem(name: "example", value: "1")]
comps.URL // https://github.com/SwiftKitz/Appz?example=1

The beauty of the example above is the absolute lack of special characters, like colons :, question marks ?, equal sign =, and ampersands &. That is all taken care of for us by this lovely class.

NSURLQueryItem

You should haven’t gotten the gist of this class from the example above. It is a very basic class that represents a name/value pair in a URL query. The cool part about this class is that it takes care of encoding all special characters that might appear in the query parameters. Hence, you should use it!

Conclusion

It is very important for developers, especially seasoned ones, to continuously research for better ways to tasks they are used to performing, like constructing an NSURL instance. New, better APIs might be available, and leveraging them might resolve hidden bugs in your code, and probably help speed your development process further.

Developing tvOS Apps

2015-12-10T00:28:04+00:00

Introduction

“Hey, Maz. We would like a port of our iOS app on AppleTV within one month max”

My response ....

Yes, lots of work is involved in porting an app to tvOS, but it wasn’t too bad all things considered. Let’s see the highlights of this experience.

RingoTV

TVJS

Before doing anything with the tvOS app, I wanted to get my hands dirty with some basic UI programming. First thing I decided to do was try out the new TVML and TVJS APIs. They seem to be recommended and are quite a streamlined way of building tvOS apps… Why not?

It was terrible idea. I easily threw a whole week worth of work out the window because… JavaScript. I dread this language… It was hard to figure your way around, everything is so streamlined and also rigid, so you have to follow some pretty dumb and strict syntax and methods to do things.

The ultimate bad thing about this was the need to rewrite all the code that deals with API calls, parses the JSON, authentication, … etc. Just. Too. Much. Work.

Dependencies

I spent a lot of time looking around for ports of dependencies we were using in the project, but without luck. We were using the monolithic RestKit as well as MagicalRecord. Both don’t support tvOS, how nice…

Then, I spent some time looking for good forks, without luck. I tried to fork these projects, update dependencies, but RestKit dependency tree is actually a hedge maze. That didn’t go well at all…

I abandoned all hope of coming up with an elegant solution at this point, and decided to just download all the sources, and drop them into the project. Using some regex, the imports were resolved, and deleting all the fat from the projects, they were up and running perfectly well withing a couple of hours. Yay!! Porting RestKit to tvOS was definitely the biggest challenge in this project, and was actually simple enough.

UIKit

This was an immediate win, and immediately felt like the right solutions. Creating a storyboard, and adding simple collection views and navigation controllers got me up and running in no time, thanks to reusing my iOS experience.

At that point, an important decision was made, which is to simplify the tvOS app as much as possible, such that it would fit one storyboard. We don’t have time for bells and whistles, so constraints had to be put in place early on to ensure the project will make it.

I would say that I only faced really few major issues, and they were pretty manageable. Keep in mind, I wanted to reuse as much of the iOS code as possible.

SELECTION

OK, so I ported the dependencies, then ported the models, and those worked great. Now, time to port the UI, or at least try to replicate it. The biggest issue here was selection. You don’t have a finger touching your UI, it’s now a mouse pad thingie!

It was actually really simple thanks to the built-in “Focus Engine”. Apple has already incorporated most of the selection logic needed, you just need to update your UI in respond to an item being focused/unfocused on.

For UITableView and UIButton, it just worked out of the box. However, UICollectionView required some work to get up and running … The work for the most part was just:

imageView.userInteractionEnabled = true
imageView.adjustsImageWhenAncestorFocused = true

Amazing! But there was an edge case where we showed a label when the image wasn’t available, so we had to handle the UILabel changing colors on focus like so (pardon the objc):

[coordinator addCoordinatedAnimations:^{

    if (context.nextFocusedView == self) {
        self.backgroundColor = [UIColor whiteColor];
        self.label.textColor = [UIColor darkGrayColor];
    }
    else if (context.previouslyFocusedView == self) {
        self.backgroundColor = [UIColor colorWithRGB:0x8C8988];
        self.label.textColor = [UIColor whiteColor];
    }
} completion:nil];

NAVIGATION

The final issue was navigation. After implementing a UINavigationController, the app kept entering background after pressing the Menu button on the remote. We needed to explicitly handle that event to make sure it controls the navigation controller instead.

override func pressesBegan(presses: Set<UIPress>, withEvent event: UIPressesEvent?) {

    if presses.first?.type != .Menu {
        super.pressesBegan(presses, withEvent: event)
    }
}

override func pressesEnded(presses: Set<UIPress>, withEvent event: UIPressesEvent?) {

    if presses.first?.type != .Menu {
        super.pressesEnded(presses, withEvent: event)
    }
    else {
        detailNavigationController.popViewControllerAnimated(true)
    }
}

We had a UISplitViewController as the main controller in the app, so these methods were added there. Then, we get the navigation controller of the detail view controller and pop there. Maybe a check should be added to make sure the navigation controller isn’t already at root, but meh. This works.

Authentication

This is the last piece of the puzzle, and honestly, still unresolved. I’ve communicated with the backend engineer that we need new endpoints for implementing authentication on tvOS, since we can’t expect the user to enter their login using the remote.

We decided to go with something similar to what digits does. That was super simple to implement on the tvOS side, but still waiting for the backend to update the APIs.

Conclusion

YES, that was pretty much it. The rest of the implementation went super smooth. The models with NSFetchedResultsController worked seamlessly with the tvOS UICollectionViewControllers. The process of hooking the models with the new UI was just a single good afternoon’s worth of work.

Swifty NSNotificationCenter

2015-12-07T12:32:25+00:00

Introduction

This is another post about Kitz. This time we will be looking at Notificationz. We will cover the motivation behind the library, and how it can easily turn your code around to be a lot more readable.

The Problem

One of the frequent pain points of iOS programming is dealing with NSNotificationCenter. It is has very “bulky” syntax, you must juggle a lot of constant string, and make sure you add and cleanup your observers properly. All this causes a lot of boiler plate code, and worst part is that it is all over the place:

class SomeClass {

    init() {
        // 1
        NSNotificationCenter.defaultCenter().addObserver(
            self,
            selector: "onApplicationDidBecomeActive:",
            name: UIApplicationDidBecomeActive,
            object: nil
        )
    }

    deinit {
        // 2
        NSNotificationCenter.defaultCenter().removeObserver(self)
    }

    // 3
    func onApplicationDidBecomeActive(notif: NSNotification) {
        // Do stuff
    }
}

As you see, three different places are littered with NSNotificationCenter dependency, not to mention this bulky API. Not only that, but you may mistakenly observe an event that sends notification on a background thread, then update your UI … BOOM. So, now you have to wrap it around dispatch_async … This is getting really ugly.

We want to do something about that…

Import Notificationz

That’s where Notificationz comes in. It is super light-weight and simple, and just aims to solve this problem. We introduce only 2 classes, and you go from the previous syntax to this:

import Notificationz

// Define your global helper only once for all files
let center = NSNotificationCenter.defaultCenter()
let NC = NotificationCenterAdapter(notificationCenter: center)

class Sample {

    private var keyboardObserver: Observer?
    private var reachabilityObserver: Observer?
    ...

    init() {
        keyboardObserver = NC.observeUI(UIKeyboardWillShowNotification) { [unowned self] _ in
            // you can write your handler code here, maybe call another function
        }
    }
}

That’s it! The code above shows you 3 important concepts:

RAII

You don’t have to do any cleanup yourself. When this object is dealloc’d, the observer variable will automatically get dealloc’d and remove the observer from NSNotificationCenter. If you like, you can prematurely remove the observer by simply setting the value of the variable to nil:

keyboardObserver = nil

Ensure Main Thread

The library provides observe and observeUI. As the name implies, observing for UI will guarantee that the notification block will be executed on the main thread. This means you can safely update your UI from the callback block.

Conciseness

Now, if you want to know what this class observes, all you do is look at the instance variables. They tell you all you need to know about what this class observes.

Also, once you observe a notification name, you immediately follow with the handler code. This makes sure that you don’t deal with “strings” that may easily break when the selector is removed for renamed.

Extra Goodness

The library offers other helpful functions that complement the observe functionality:

NC.add(obj, selector: "call:")  // normal add observer
NC.observe { notification in }  // observe using blocks
NC.post("Ten-hut!")             // post a notification
NC.remove(obj)                  // remove from nsCenter

Conclusion

If you want to make your code look pretty again, take a look at this library, and consider dropping it in. Contributions welcome, and there are some other pretty cool libraries as part of Kitz

Playing With ReactiveCocoa

2015-12-05T02:10:51+00:00

Introduction

ReactiveCocoa is a long standing library, it’s by no means something new. The cool kids were already high on ReactiveCocoa since the Objective-C days. Alas, it wasn’t something I decided to try before today.

Through this article, I’d like to cover my initial impressions in using ReactiveCocoa with Swift, especially because of the insane potential that it has to dramatically streamline apps.

Reactions

I will not sit here and talk about reactive programming and pretend I am an expert, because I am not. I will simply present a metaphor that I like to keep in mind, then share a bunch of useful links on the topic. I hope you are familiar with circuit boards, because that will be the basis of our metaphor.

When designing a circuit board, the usual parts we tend to use often are the power source, resistor, capacitor, inductor, … etc. The power source would be the “producer”, such as a battery, and the resistor would be the “consumer”, such as a lamp. Then, we have other parts that incur various effects. For example, we can think of a capacitor as a gatekeeper, since it only allows alternating current to pass, and serves as open-circuit when full.

Now, you as the circuit designer, would probably start with the power source, and decide how the current should flow, doing any processing as necessary. For example, you may add two power sources, and only switch on the light if both are active.

That’s pretty much it, as far as I know. Reactive programming has “signals” instead of power sources, and you observe these signals, react to them, apply various processing routines around them, and something else happens as a result.

See ReactiveX See wikipedia

Functional Reactions

Now all that is nice and dandy, but how about a practical example with Swift and ReactiveCocoa 4.0 alpha4?

The example we will take a look at is something I am actually using in a shipping app, so it should be practical and useful. I admit, though, it’s the same example you’d see everywhere else… Taking user input, validating, then reacting with colors and enable/disable button, specifically for login and signup.

What does this look like?

What about teh codez?

First, we simply store the actors in our little reactive play in local variables. We will see later this will prove useful for many things, like breaking retain cycles.

Note how we store NameValidator.validate into an array. This is simple a static method that takes in a String and returns a Bool. This means, validators is an array of functions that all take a String and return a Bool.

// Prepare actors (avoid retain cycle)
let button = signupButton

let fields = [
    firstnameField,
    lastnameField,
    mobileNumberField,
]

let validators = [
    NameValidator.validate,
    NameValidator.validate,
    MobileNumberValidator.validate,
]

Now, we prepare the signals we would like to observe and process. The signal we are interested in is the rac_textSignal. It emits a signal whenever the text property of UITextField changes.

We create the signals by first zip-ing the field with its validator, the iterating on those by getting the text signal and mapping its result to the validator. This is the first arrow in the graph above. The signal emits a String each time text changes, then we pass that to the validator, returning for us a Bool.

// Prepare signals
let signals = zip(fields, validators).map { field, validator in

    field
        .rac_textSignal()
        .map { validator($0 as! String) }
}

Now, we simply add our process block that will set the color based on whether the validator returned true or false.

We iterate a zip-ed fields and signals this time, and for each signal we add a process block that will set the field valid property to the validator result. This is actually a UITextField subclass that has a valid property, which upon being set, updates the colors.

// Apply color subscribers
zip(fields, signals).forEach { field, signal in
    signal.subscribeNext { field.valid = ($0 as! NSNumber).boolValue }
}

Now, this may be achieved in a better way by including this code in the UITextField subclass, then assigning the validator to the UITextField itself. That’s not a bad approach, I might go with that actually.

Now, let’s see how we can reduce the three signals emitted from the textfields into one signal using &&. We only enable the sign up button if all textfields validate, after all.

First, we “combine” the signals, which simply puts all the latest signal results into a RACTuple. Then, we “normalize” the RACTuple using allObjects. After that, we cast that array to [NSNumber], and reduce that using &&, meaning all signals must be true. Finally, we add our custom process block, which updates the button enabled state to the reduced result we just computed!

// Apply button state subscriber
RACSignal
    .combineLatest(signals)
    .map { ($0 as! RACTuple).allObjects() }
    .map {
        return ($0 as! [NSNumber])
            .map { $0.boolValue }
            .reduce(true) { $0 && $1 }
    }
    .subscribeNext { button.enabled = ($0 as! NSNumber).boolValue }

I really have no more things to say, it almost works like magic. All this is a one time setup in viewDidLoad, which just makes life way easier.

Conclusion

I always try to be honest and blunt about my thoughts, so here is the last bit. I would probably never use ReactiveCocoa in a long term project that I need to personally maintain. Dealing with Swift flux is painful enough, and thus apps that I look to maintain have absolutely minimum external dependencies, especially enormous libraries that I can just fork and build my own version of.

Then, why did I use it? I am working for a client on a simple app, and the goal here is to cut costs and roll out an app ASAP. This app won’t be doing any insane UI/UX features, it’s all about building a small footprint application to validate the market. Hence, ReactiveCocoa saved me a whole bunch of extra code I would otherwise had to write and prolong the project.

P.S. I really don’t get developers who refuse to check in the project’s podspec into the main repo. Something about not willing to maintain it, or incur the burden … I mean, c’mon, just do the whole iOS community a solid here and check it in…

Freelancing Stories

2015-12-02T11:24:16+00:00

Introduction

It wouldn’t be unusual to be sitting at some day job thinking… Wouldn’t it be great to be one’s own boss? Taking vacations anytime, no one pestering you about what you wear or when you wake up … Sounds like the perfect lifetime plan! But how…

Of course! Why not try freelancing? You pick your own projects, finish work at your own pace, heck, you get get to decide what your salary is gonna be. You can’t possibly go wrong there? .. WRONG.

Negotiation

Before any project takes place, you must actually find a project. If you are a software developer, you simply don’t have a problem there. Period. Clients are everywhere, with so much variety, there is literally a client for any weird scenario you may think of. That’s why I decided to skip the discovery phase and start with the negotiation phase right off the bat.

Lesson 1: People Matter

Negotiating purely on the project scope and pay spells out disaster for you as a freelancer. First think you should care about are the people you’ll be working for.

One of my clients I worked for wanted a pretty simple and straight-forward iOS app. They had absolutely no clue how developing iOS apps work, and I thought to myself: “That’s great! I get to set my own rules and standards, and teach them along the way”. We agreed on the price, money was paid on time and all, and the project started…

As the project progressed, I first kept working on the UI side, throwing suggestions at the client, and the client was happy. The client was seeing progress, listening to my feedback, and only synced up with me once a week.

By the end of the first sprint, I showed the client a complete basic app fully functional, but without backend integration. All the data in the app was dummy data, but at least the UI implementation was almost perfect. The client sent back some minor feedback regarding the UI demo, and decided to move forward with the project and start integrating the backend … This is where things went south.

As I started integrating various endpoints, I discovered many problems with the backend API design. It was flawed. I patiently worked with the backend developer to amend these flaws, and trying to integrate whatever I can with the app itself. At that time, the transition between Xcode 6 and 7 happened, as well as my discovery that CoreData and Swift don’t work well with each other.

Progress wasn’t being made as fast as it started thanks to these problems, and then out of nowhere, the client got worried and discouraged. I tried to explain how the problems affecting development were slowing the progress, but due to severe lack of communication, the client just made up theories about this project, and decided to end the contract.

This was the best possible outcome for me, I was starting to hate the project, but still. I had worked hard to put this together and didn’t like it ending that way. So, I started thinking back, where did it all go wrong?

It was the negotiation phase. If the client knew a little bit about the issues I was going through, they would’ve appreciated the effort put into making this app happen. I guess trust plays part, as well. The client didn’t trust anyone, and even complained about the designer not doing his job… It was only a matter of time I would run into this issue, and I did.

Lesson 2: Price

Whatever you do, never work for a fixed price. Never. No exceptions.

There are literally zero clients that will come to you and say “Hey, my project is complicated, and I appreciate any effort you put in”. Instead, all their projects are “simple” and “straight-forward”, should be done in less then a few months, and they have all the details figured out… You’re looking at the wireframes going, “Yeah… that’s a scribble on a napkin, what am I suppose to make out of that?”

The solution to that is actually quite simple. Don’t work for fixed price, ever. For your sake and the sake of the client, working for fixed price means one will be cheating the other. It’ll either be the developer doing a crappy job and demanding the whole pay, or the client making unreasonable claims about the scope as they envisioned it, demanding you see it through (even if it takes years). Really, you don’t want that.

If you are paid an hourly rate, it no longer matters how things turn out. If you are are not holding up your end of the bargain, your payment is cut. If the client wants to keep making demands, they have to start shaking them pockets. It’s simply the most fair setup I found.

Conclusion

Well, look at that! Just discussing the negotiation part took so much to cover, and yet more parts are to be discussed! I’ll leave this here for now, but will hopefully post more on this topic later.

iOS or Android Remote Logging

2015-11-30T22:42:15+00:00

Introduction

I have highlighted in a previous post that I’ll be writing a topic about Sentry. The reason I said that was because I was in the process of building a remote logging service for an iOS app. Ultimately, it took less than two hours to implement.

Here is the scoop: Sentry is just impossible to deploy for a n00b like me. It is dependent on nginx, django, redis, and postgres… Can you believe that? I just want to get a logging server for a simple app, and I have to learn and deploy all these services? I didn’t have the luxury of time, so a pivot was inevitable.

Firebase came to the rescue. I was able to read their docs, learn how it worked, design and implement the remote logger in the iOS app, all in less then 2 hours. That’s how great Firebase is.

Business Needs

First off, let me shed some light on the situation here…

The App we are talking about is a B2B native iOS/Android application. It is deployed to a handful of businesses in order to streamline a certain activity of theirs. That’s it, let’s try to break this paragraph into requirements.

The fact that the application is B2B puts it at a whole other level than regular B2C apps. B2B implies higher standards, less error tolerance, and extremely high support expectancy. If the app crashes while they’re using it, they expect you to be on top of it ASAP, with little intervention from their side.

Then, there is the fact that we have native apps out there. That puts us at a disadvantage in terms of reachability and maintainability, since it’s almost impossible to quickly and accurately monitor user activity and provide support in real-time, like web apps do… But, we can actually address that.

Finally, we are only deploying to a handful of businesses. This eliminates the need of complex data arrangements and needs. For now, it’s important to me mean and lean!

Detailed Monitoring

The problem we would like to address today is: Monitoring.

After submitting the application to the AppStore, and our users downloading the apps, we would like to quickly find problems and be alerted, if any, before the users even reach out to us. This will help our company’s image in being professional and on top of problems, as previously highlighted.

The parameters needed to make sure this works are:

The User
Log Type
Some Info

We really need to log events per user, have distinct log types in order to filter out the trace logs from the actual errors, and finally some info associated with these logs in order to figure out what is happening!

Firebase Logger

That is how Firebase Logger was born. I apologize in advance for the crappy Obj-C code, but this is an old project:

/** A simple logger that sends all the logs to firebase
 */
@interface NSHFirebaseLogger : NSObject

- (void)logSuccess:(NSString *)message, ...;
- (void)logInfo:(NSString *)message, ...;
- (void)logWarn:(NSString *)message, ...;
- (void)logFail:(NSString *)message, ...;

@end

The interface of the logger is nothing exciting at all.. It simply takes in a message format with arguments to construct the actual log message for a certain log level. The actual log method is where things get interesting…

- (void)_log:(LogLevel)level format:(NSString *)format args:(va_list)args
{
    NSString *log = [[NSString alloc] initWithFormat:format arguments:args];
    log = [NSString stringWithFormat:@"[%@]: %@", LogLevelLabel(level), log];
    ...

The beginning of the method, we simply want to build the full log message, including the log level. So the code above simply spits our something like:

[WARN]: User decided to actually press the delete all data button

    ...
    Firebase *firebaseLogs = nil;
    NSDictionary *deviceInfo = [NSDictionary deviceDetailsForLogin];
    NSHUser *user = [NSHAppManager sharedManager].account.loggedInUser;
    ...

Now it gets interesting… We first prepare a Firebase reference, we will see what that will do later on. Then, we use a convenient NSDictionary call to get the device info, like the device model, MAC address, … etc. Finally, we try to query for the logged in user from the account manager class.

    ...
    if (user) {
        firebaseLogs = [self.firebaseRoot childByAppendingPath:user.username];
    }
    else {
        firebaseLogs = [self.firebaseRoot childByAppendingPath:deviceInfo[DeviceIdKey]];
    }
    ...

About the user, if there is a user logged in, we want the firebase log tree to start with that user’s username. If that is not possible, then use the device ID, which we log in the backend anyway, as the root of the logs.

As you can see here, Firebase is a simple and elegant object that is like a key value “tree”. Keys and values can be nested in a tree manner, as we will see in the end. So, taking the root Firebase object, we start to build the tree by creating a child using the appendPath method.

    ...
    firebaseLogs = [firebaseLogs childByAppendingPath:deviceInfo[DeviceModelKey]];
    firebaseLogs = [firebaseLogs childByAppendingPath:@"logs"];
    firebaseLogs = [firebaseLogs childByAutoId];    
    ...

Then, we simply continue building this tree by adding 3 more children. The first is the device model, just in case the user has multiple phones. I figured it’s quite rare for a user to have two devices of the same model, so this should be ok.

Then, we add a hard-coded “logs” node to contain the keys for all the logs. We do that by immediately adding the childByAutoId, which simply creates a unique id for the log we are storing, right under “logs”.

    ...
    [firebaseLogs setValue:log];

    NSLog(XCODE_COLORS_ESCAPE @"%@ %@ " XCODE_COLORS_RESET, LogLevelColor(level), log);
}

Finally, we assign the value of this node that we reached to the log message. We also print the log to the console for debugging purposes. I should probably wrap it with #if DEBUG, though.

Now, let’s see what this handy work of ours gives us:

Isn’t that just dandy! This is the default firebase web viewer, and you can see how neatly the logs are structured, and how easy it is to quickly dig into them and find issues for a particular user.

Conclusion

The best part is, it doesn’t have to stop there! Adding a cron job to process the logs and give alerts or whatnot wouldn’t be a bad idea. Also, it could provide a better log reading experience. The point is, the base is taken care of by Firebase, you just have to use your imagination from hereon onwards.

Oh My Zsh!

2015-11-28T15:18:38+00:00

Introduction

Hardly any developer can go about his career without interacting with the terminal on daily basis. Many tools continue to be developed as command line apps, like fastlane, Jekyll, and many more … With all that being said, we need to make terminal more usable!

Recently, I’ve been ssh-ing into an Ubuntu server that defaults to bash, and I quickly realized how my terminal life has been completely turned around thanks to zsh. Here are a few examples of why it is just pure awesomeness.

Before I begin, let me just say that I have immediately installed Oh My ZSH right after installing zsh itself, so I am not sure which features are specific to zsh and which are added by the plug-in.

Z Shell

When mentioning the features, I guess I’ll sort them in terms of the most important to me, down to the most “meh”. It’s purely subjective, but here we go:

Timelapse

I can’t get enough of this feature. You simply type a command, then hit the up arrow key, and BAM! You are cycling through your history of that specific command:

% ssh-add ↑
% ssh-add ~/.ssh/github-terminal

This feature saved me countless (accumulated) hours. From running py.test, adding ssh keys, all the way to simply removing Xcode’s derived data folder:

% rm ↑
% rm -Rf /Users/mazyod/Library/Developer/Xcode/DerivedData

It is by far my best feature, but the rest of the features are pretty kickass as well.

Cycling

If you are a lazy typer like me, then you’d feel for me when I say that nothing is more annoying than bash continuously blocking [Tab] completion because many completions exist!

$ ss⇥
SSH_AUTH_SOCK  ssh-keygen   
ssh            ssh-keyscan  
ssh-add        sshd         
ssh-agent      sso_util
$ ssh-⇥
ssh-add      ssh-keygen
ssh-agent    ssh-keyscan
$ ssh-a⇥
ssh-add    ssh-agent

Annoying!!! How does zsh solve that? Well, if you hit double tab, it shows a selection highlight, so you can cycle through the options and choose the command you like!

This is just so convenient…

Auto-Correction

Once you start typing a command then hit Tab, it can correct your typo automatically, and does so in many ways!

CUSTOM AUTO COMPLETION

With the right plug-ins from oh my zsh, you can have custom auto completion for your favorite commands! Let’s see git for example:

% git⇥
add       -- add file contents to the index
bisect    -- find by binary search the change that introduced a bug
branch    -- list, create, or delete branches
checkout  -- checkout a branch or paths to the working tree
clone     -- clone a repository into a new directory
commit    -- record changes to the repository
diff      -- show changes between commits, commit and working tree, etc
fetch     -- download objects and refs from another repository
grep      -- print lines matching a pattern
init      -- create an empty Git repository or reinitialize an existing one
log       -- show commit logs
merge     -- join two or more development histories together
mv        -- move or rename a file, a directory, or a symlink
pull      -- fetch from and merge with another repository or a local branch
push      -- update remote refs along with associated objects
rebase    -- forward-port local commits to the updated upstream head
reset     -- reset current HEAD to the specified state
rm        -- remove files from the working tree and from the index
show      -- show various types of objects
status    -- show the working tree status
tag       -- create, list, delete or verify a tag object signed with GPG

WRONG COMMAND

% xcodeguild
zsh: correct 'xcodeguild' to 'xcodebuild' [nyae]?

Not only does it correct it for you, but also allows you to hit e and edit the command, just in case it didn’t get it right.

MIND READER

zsh ships with a mind reader, really! Look:

% ls
blog          untiy3d-game  upcoming-book
% cd game⇥
% cd untiy3d-game/

I DoNt cArE

As a final note, zsh doesn’t torture you for not getting the CAPS right. It will happily accept wrong caps for files, and correct that for you.

% cd doCuMent⇥
% cd Documents/

ZMV

I am not sure where this command comes from, but it is just super for renaming files. All you have to do is autoload it, and then use it. For example, let’s say we have a bunch of png images we want to append @2x to the end, before the extension:

% autoload -U zmv
% zmv '(*).png' '$1@2x.png'

This is not all … If the rename will mess up your files by overwriting each other, it will detect that and abort the whole operation, leaving your files intact! In case you didn’t understand, let’s see the following example.

Here, I inaptly tried to rename all the images in an Xcode catalog tree to “oops.png” in the root directory. Normally, this command would mean I lose all the images, and only get a single image “oops.png”, corresponding the the last image the command runs on. Not zmv. It gave me a readable error, and aborted the operation. I checked git, nothing changed at all!

(master)⚡ % zmv '(*)/(*).png' 'oops.png'
zmv: error(s) in substitution:
button.imageset/button@2x.png and button.imageset/button.png both map to oops.png

Themes

YES! Zsh allows you to install custom themes shared by other awesome people, and the choices are just too good. So many different themes, it’s hard not to find what you are aiming for.

My choice was tricky, yet I still found the exact theme I wanted. I need something that shows the cwd on the right, the colors to be according to SMYCK color scheme, displays the active git branch, and nothing else. It must be minimal .. and I found it, the wezm theme:

ZSH_THEME="wezm" # norm also good

Misc

Here is a dump of more features that collectively make terminal a hell lot more usable.

NUKE CONFIRMATION

I am not worried about typing sudo rm -Rf * anymore.

% rm -Rf *
zsh: sure you want to delete all the files in /Users/mazyod [yn]?

Lavish Ideas

2015-11-27T00:33:33+00:00

Introduction

While thinking for a topic to write about, I realized that just the fact of choosing a topic is a whole post in itself! Recently, I’ve been discovering cool new technologies, and I just can’t wait to write about them .. I’ll post them here for remembering that later.

Ideas Are Dime a Dozen

Sentry + iOS

I’ve been integrating Sentry with an iOS app these days, and it’s a really rewarding experience. I’ve assumed charge of the whole stack, from deploying Sentry on AWS, all the way to writing the exact logs in the iOS app itself. Will hopefully post about that as soon as its done!

Post published!

AppleTV

Another thing is developing an Apple TV app from an existing iOS app source. AppleTV SDK is pretty good, but there are some really nice (and really weird) differences. I’ll hopefully highlight that experience once it is done, as well.

Post published!

zsh

zsh is just just simply the best thing that happened to my terminal experience hands down. In terms of overall benefit to my workflow, it easily out-shines Homebrew, cask, and any other tool I’ve ever used .. COMBINED. I just have to write about how awesome zsh is .. In fact, I’ll start drafting this post right now, and just fill it up as I keep using zsh everyday.

Post published!

On Being a Freelancer

I’ve worked on two freelance projects in the past few months, and I believe there are some really good lessons I’ve learned that could be worth sharing. It’s unfortunate that freelancing isn’t what we imagine it to be…

Post published!

WatchOS Apps

Of course! I’ve recently shipped an app with AppleWatch support, including glances, complications, and time travel! Definitely interesting topics to be discussed on that front.

Kitz

As I build more bits and pieces of Kitz, I keep learning more and more important lessons I’d really like to discuss in depth on this blog. I’ve already posted about one, but there is way more stuff to be posted.

Post published!

Conclusion

Software development is just a single field in a whole ocean of knowledge, and yet it is an ocean itself in its enormity… May be overwhelming at times, but if you imagine it as some sort of RPG, it’s always exciting to go about discovering new dungeons! Always more fun with a party, as well!

Hacking With Jekyll

2015-11-25T18:03:49+00:00

Introduction

For those of you who are new here, welcome! For the rest of the old time readers, yeah, the blog has been completely redesigned .. From scratch. I hope the new design is a step forward at least, and in this post, I’ll be covering the transition experience.

Where Were We?

The previous blog was built on top of Octopress and using a heavily modified octoflat theme. That is to try and say … It was ultra-bloated.

Octopress

Octopress was amazing .. As long as it was actually working. It had many convenient wrappers around Jekyll, uses Ruby rake to do stuff, and wasn’t so bad all things considered. The only problem was, it was complicated.

Octopress was using Jekyll plugins to its fullest. Also, I was using it pre-v3.0, so it was tightly integrated with git. That made upgrading octopress an unpleasant chore.

Most annoying crap I dealt with was compass. It just didn’t work most of the time, hard to configure, and full of bugs. Thankfully, it’s removed in most recent octopress releases.

Another pain point was generating the blog from source … Since I was using lots of plugins, I couldn’t make github generate the website for me, so I had to build it locally, then push the result. Generating the blog took about 10 - 15 mins! That was crazy .. I know I have 200+ posts, but still! This was really annoying, especially if I had made some sort of syntax error or want to preview a post.

Finally, the elaborate directory structure and options were overwhelming for my needs, hence it was time to be minimalistic.

Octoflat

I shouldn’t talk about this .. Countless times I get hollered at by Jim because something broke, like pagination or image size on Android. Even more annoying was tweaking the design a bit, and watching all hell break loose. It is just a very bloated theme for my simple needs. How not when it includes bootstrap, flat-ui, and a million other styles, all mushed together. The !important tags were countless, lol.

All things considered, I really appreciate the author’s efforts in open sourcing it. It served me well these past few years.

Experience

I noticed that the reader feature on iPhone wasn’t available, which is a huge missing feature for readers. Another problem was the overall design of the blog. I hadn’t thoroughly thought about UX back then, so colors, fonts, layouts were all just a huge letdown.

Where Are We?

The new blog is so simple, I don’t even have to break it down into sections. It’s a pure Jekyll powered blog with minimal changes on the default theme. Yup, all I did was tweak the default theme a bit to achieve the current look, without adding a single dependency or framework.

The blog finally generates in just 7 seconds, compared to the original 15 minutes, I’d say that’s a significant win. Also, all I have to do is push the changes to github, and github will build the blog for me! Yatta!

Admittedly, I had to rewrite all the script I had, but that’s ok! Now that I have full control over the template created for a new post, I went ahead and created a much more sensible default template for my needs. All it took was a couple of minutes, really. I’ll probably blog about the new workflow some day!

Last but not least, now I have the capacity to maybe do A/B testing, see which changes readers like more and so on, since making changes is so easy!

Conclusion

I am in love with zsh. Seriously, if you’re not on that thing, you’re missing out, because this shit is thoroughly good!

One More Time

2015-11-23T16:08:12+00:00

No introduction, no conclusion, just pure anticipation of the new blog design to be rolled out. Yes, the blog shall be redesigned and relaunched one more time. I am spending my free time on the redesign rather than blogging, so here is something to generate some hype:

Coming soon

Wrangling Autolayout

2015-11-21T20:30:58+00:00

Autolayout Ambiguity

I’d like to share today a short story about autolayout. Before I get into that, however, I’d like to give some background regarding my experience with autolayout for the sake of helping you better relate to the upcoming story.

Auto-layout was just an unusable nuisance for me when it came out on Xcode 4 for iOS. The first step after creating any XIB file was to turn it off. It’s not like I didn’t try it .. I did, but back then, Xcode would crash, layout would easily mess up your views, … etc.

As I progressed in my career, I came across an use case that inevitably has to use autolayout (unless I want to lose my mind). That use case was Right-to-Left (RTL) support for the Telly app. In order to properly support RTL layout without writing a million tedious layout code everywhere, I had to use autolayout with its automatic layout correction. Luckily, that worked really well, since the views I needed to add support to were all properly broken down and created in XIB files. enabling autolayout and setting things up was child’s play.

Those days are over now, and I’ve recently started doing some fresh projects and experimenting with Storyboards, Segues, and more Autolayout. Here is the weird part: I’ve built two apps that started with Storyboards and Autolayout, and ended up deleting them and using manual layout (layoutSubviews())! Why?

I simply didn’t have enough time to step back and see exactly where my layout was breaking. I remember quickly running into problems and quickly switched to something I am proficient at, manual frame calculation… But, the sad part follows.

Due to using manual frame layout, I lost considerable amount of time on those projects in total. Writing manual frame layout is very easy, but extremely tedious and fragile. I had to take the time to learn Autolayout properly, and make it work.

Master of Autolayout

Let me go ahead and say this: The master of autolayout is PureLayout. Let’s see why…

I recently started yet another client project, and true to my earlier conviction, I’ve went full-blown Storyboard + Autolayout. What made matters even better was that the client was OK with supporting iOS 9+, pretty suh-weet, for only one reason: UIStackView. Let’s leave that for another day .. For now, autolayout!

So, I was happily building the project, using autolayout all over the place, until I hit my first serious hurdle. The client wants to implement an intimidating looking transition. I can’t post the assets here, but try to imagine it with me:

“There is a view with a stack of cards in the middle somewhere. When a card is tapped, it expands to fill the screen, and then scales back down when closed.”

I can’t simply manipulate the view’s frame right away, since it is embedded within the view hierarchy. So, the approach I thought of was the following:

Remove the view, and add it as a direct subview of the key window.
scale up the view

Sounds pretty reasonable … But this is where autolayout haunted me.

After writing the code above, it simply didn’t work. I would end up with an empty screen … Debugging the view hierarchy using Xcode and Chisel, I can see that the transitioning view’s frame always ends up being all zeros. Why??

The experimentation began … I replaced it with a dummy view, it worked like a charm. I replaced it with a new copy of itself, and worked, but a bit funky … I am missing something! Finally, setting translateAuto.. to true made the view at least appear .. Oh no, an autolayout problem! Do I really want to bother with that? Should I just stick with what I know? … Time to start learning new stuff!!!

So, I add PureLayout into my Podfile, created a container view for the transitioning view, and used autoPinEdgesToSuperviewEdges. That was the end of it … The solution was insanely simple and worked like a charm! I guess the takeaway here is to either fully embrace Autolayout, or not. If you decide to embrace autolayout, just do yourself a favor and get PureLayout, it’s kickass.

Conclusion

I have mixed feelings about this .. I am glad and relieved for finding this solution, but also distressed at the fact that I’ve missed it for so many other projects. Well, our only choice ever in life is to look forward!

Generic Constraints with Optionals

2015-11-20T13:13:34+00:00

Introduction

Recently, I’ve been spending a lot of time polishing and testing a collection of Swift frameworks and releasing them on github. You can see the website here.

The lessons learned from this experience were unbelievable. There is way more to Swift that I could have ever imagined. The fun part is, each kit in the project comes with its own design problems, and hence requires different approaches and tapping into more amazing Swift features.

For this post, I’ll be going over a design challenge I faced when writing Storez.

Conformance

So, Storez is a a very flexible Key/Value store for Swift that allows end-developers to easily define their keys in a type-safe manner (with custom behaviors). The biggest challenge faced when doing that is the “type-safe” part. How do you ensure that?

Well, let’s take the UserDefaultsStore as an example. It uses NSUserDefaults as an underlying persistence store. NSUserDefaults doesn’t support every single type of object, only a certain subset. How do we make sure that objects to be stored can be serialized to those types?

Protocol conformance is the key! By defining the following protocol:

// inherits from class, since we need to cast it to AnyObject
protocol UserDefaultsSerializable: class {}

We can then “tag” the classes which are supported by the store:

extension NSNumber: UserDefaultsSerializable {}
extension NSDate: UserDefaultsSerializable {}
extension NSData: UserDefaultsSerializable {}
...

Now, we write two simple get and set functions to access the store:

func get<V: SerializableType>(key: String) -> V? {
    return defaults.objectForKey(key) as? V
}

func set(key: String, value: SerializableType?) {

    defaults.setObject(value, forKey: entry.key)
    defaults.synchronize()
}

The above code is a complete example of providing a safe API on top of NSUserDefaults… But, adding custom type support would be nice, won’t it?

Convertibles

This time, we use “conformance” to provide a set or rules for custom objects that want to store themselves in the key-value store. Here is the one written in Storez:

/** Other types can conform to this protocol to add  support.
    It simply requires the class to convert to and from one of
    the supported types.
*/
public protocol UserDefaultsConvertible {

    typealias UnderlyingType: UserDefaultsSerializable

    static func decode(value: UnderlyingType) -> Self?
    var encode: UnderlyingType? { get }
}

This is really interesting, isn’t it? First, we provide a generic type requirement, which is furthermore constrained to a UserDefaultsSerializable type. Then, the conforming type needs to provide two function implementations, one for decoding (de-serializing), and one for encoding (serializing).

Note, even an enum can easily be a conforming type! Since it is a pure Swift protocol, there are absolutely no limitations to which type can add support.

One last caveat remains .. What about non-optionals? If you look closely at the get function, it always returns an optional. What if we wanted to support non optional values, that return some default value when the persisted value doesn’t exist? This presents its own set of challenges, since we now have to provide two separate flows for option, and non-optional types.

Read on…

Get Under Optional Skin

Let’s first see how we provide the two get variations:

public func get<E: EntryType where E.ValueType: SerializableType>(entry: E) -> E.ValueType {
    return _get(entry.key) ?? entry.defaultValue
}

public func get<E: EntryType, V: SerializableType where E.ValueType == V?>(entry: E) -> V? {
    return _get(entry.key)
}

So, what do we have here… The first method is the “nonnull” variant which always returns a non-optional value, while the second method simply returns a nullable value.

The key difference between these two methods is the way we check to make sure that the value conforms to UserDefaultsSerializable type. In the first method, it’s pretty straight forward. We immediately assert that the ValueType of the EntryType is a conforming type.

For optionals, it was whole journey to reach this final, succulent form. When you look at it now, it feels very natural and straight-forward! It basically says: Define two generic types, one is the EntryType and another is a value type that conforms to UserDefaultsSerializable. Then, simply make sure that the EntryType’s ValueType is an optional of the conforming value type!

Conclusion

The power of Swift is just mind blowing. I’ve worked quite a bit with C++ generics, and they are much more flexible, but no where as safe and well defined as Swift’s.

UIApplication Trap

2015-11-18T17:03:03+00:00

Introduction

To all my readers out there .. An apology is in order. I have been blogging non-stop for a while, but there was a very unsettling series of sudden pauses recently! I promise it wasn’t due to lack of motivation nor interesting topics .. It was a simple matter of prioritization.

Working on about 6 project at the same time is just not something I’ve been used to, since I usually put my all into one single thing. Believe me when I say, spreading out like this is much more rewarding, since all your eggs are no longer in one basket.

In all cases! With that lifestyle, returning to Habitica was a no-brainer. With Habitica back in the picture, its time to level up like crazy!!

The Wrath of the Singleton

Singletons are very handy, and I don’t even remember how many times I’ve blogged about them … Unfortunately, I ran into one of those cases where it just kicks your ass.

The code I have in my app is like this:

class NotificationsMigration: MigrationTask {

    func shouldExecute(previousVersion: Int) -> Bool {
        return previousVersion < 2
    }

    func execute() {
        UIApplication.sharedApplication().cancelAllLocalNotifications()
    }
}

So, in my app I can easily define subclass of MigrationTask in order to add code that should be run only once when migrating from a previous version.

With that said, this migration code is suppose to cleanup previous local notifications because the new version of the app handles notifications in a different way. If I don’t do this, users migrating from the old version will experience double notifications (ones that are setup previously, and new ones setup by the new version).

Well, that looks nice and dandy! There is just no way this code has a bug, right? There is no need to test it, right? Right, I thought! .. and boy was I mistaken…

After publishing the app to the AppStore (Yes, it’s that bad), I got a horde of users coming at me complaining about seeing double notifications firing! What?! Why?! How?!!?!

I had no choice but to put a break point there, and see what was going on. Was the code even reached? Yes, the code reaches the statements and executes perfectly?? WHY?! I brushed it as a glitch in iOS, and submitted another update with other fixes … No, that’s not the end of it.

While refactoring another part of the app that dealt with notifications as well, I started testing the new code. After running the app, I noticed that notifications were not being scheduled at all … The code was this:

application.scheduleLocalNotification(notification)

The app variable was captured by calling UIApplication’s sharedApplication() accessor as soon as the object is initialized. That was really weird, why won’t it get scheduled?

By doing some excessive debugging and investigative work, I noticed that this object is initialized before:

UIApplicationMain(Process.argc, Process.unsafeArgv, nil, NSStringFromClass(AppDelegate))

Yes, I override main.swift, and do some stuff there necessary for language settings, and somehow this object initialization slipped there! To my pleasant surprise, querying for sharedApplication at that point returns some dummy object! Not even nil! The app continued to “not crash”, but doesn’t function properly >_<.

After suffering from this singleton wrath, I appropriately moved most of the code that executes before the UIApplicationMain call into the app itself, somewhere where it executes after applicationDidFinishLaunching is called.

Conclusion

Seemingly simple tweaks in the code are still considered “rewrites” if they change the behavior of the program. Rewrites are hard, because they need to be tested thoroughly, and testing itself is hard. That is probably the topic of the next post, so stay tuned!

Where is Maz?

2015-11-07T13:35:42+00:00

🇯🇵 Tokyo

Where Next

Just staying in Tokyo for a while :D

About This Page

A friend said for fun:

We need a WhereIsMaz.com site, we never get a hold of you!

Well, I decided to make it anyway, so here I am!

The reason for this is my current lifestyle as a digital nomad means, I am just constantly traveling, and never really making up my mind on the next destination till one week before .. At most.

App Submission Nightmare

2015-11-01T10:36:47+00:00

Introduction

It’s that time of the year where I need to submit an app to the AppStore. This time around, I had a completely new setup that I’ve never attempted to ship before. As expected, since it’s a new setup, it came with its own submission error variety.

The Setup

My setup is quite complicated, and neatly sits in the edge cases of Xcode project setups.

The Workspace

The project workspace and directory structure looks something like:

Workspace
- Framework1.xcodeproj
- Framework2.xcodeproj
- …
- Framework10.xcodeproj
- MainAppProject
  - Framework1.xcodeproj
  - …
  - Framework10.xcodeproj
  - MainAppGroup
  - MainAppUnitTests
  - MainAppUITests
  - TodayExtensionGroup
  - AppleWatchExtensionGroup
  - AppleWatchAppGroup
- Pods
- Pods

Here is what we are dealing with here:

We have 10 Framework projects + 2 cocoapods projects + 1 iOS app project
Everything is Swift based, except Framework1, which is Objc.
The main app depends on cocoapods as well as all included framework projects
One of the framework projects also depends on cocoapods
The main app itself has 6 targets
1. iOS app
2. unit tests
3. UI tests
4. today extension
5. AppleWatch app
6. AppleWatch extension

Submission and Dominance

At least, this is the game Xcode was playing ..

Code Signing & Provisioning

First problem that every iOS developer on earth faces came jumping at me:

Code signing in today extension doesn’t match provisioning profile

I checked the today extension target, and it had a bundle ID that seemed different than the one I had … Instead of fighting Xcode, I launched sigh and typed:

sigh renew -a com.ArabianDevs.PrayerTimes.today-extension --force

Warning Galore

For some reason, Xcode was throwing 100s of warnings about “stripping sybmbols”:

warning: skipping copy phase strip, binary is code signed

First search through stackoverflow and google led me to disable “strip debug symbols during copy” for release builds as well as debug. After researching more about it for this article, I read an answer from a TestFlight engineer that this bloats the app by about 30% - 50%, which is insane. So, better just ignore this harmless warning.

Archiving

Finally! The archive is complete, and showing up in Organizer… Wait, why is it listed under “Other Items”, and not iOS apps?! I can’t submit to the AppStore this way :(

Most answers on google point you to making sure that you set “Skip install” to true for all subproject frameworks and libraries… I checked, and double checked all 10 frameworks several times, they were all correctly configured.

I had no idea what was going on, but the only thing I could do is inspect the generated archive, so I did. Opening it revealed that in the Products directory, there was “/Library/Frameworks/Framework1.framework” … Why the heck was it there??

Coincidently, Framework1 was the only ObjC framework in the lot, so that aroused my suspicion. However, after looking more closely at the project, I realized that it also utilizes Cocoapods! I recall reading that Cocoapods had issues with archiving frameworks pre version 0.39.0 .. Checking Cocoapods for that framework, and true enough, it was built with cocoapods 0.38.2.

Thankfully, all it took was another pod install, and cocoapods took care of upgrading the project. Archiving the app again made it appear under iOS Apps! Yay!

Submitting

So, it’s finally time to hit that Submit button in Organizer, and hope for the best … That didn’t go far at all. After choosing the team and hitting submit, I immediately ran into this error:

The archive did not contain DVTFilePath: X: ‘X/Library/Developer/Xcode .. X.xcarchive/BCSymbolMaps .. .bcsymbolmap’> as expected.

That was a really weird error to get .. I remembered that my binary isn’t bitcode compatible, so I unchecked that … No dice. Searching online led me to this post. Surely enough, after unchecking the “include app symbols”, the error went away .. Phew.

Validation

After the app started validating with iTunes connect, that is when things got rowdy. I got 4 annoying errors, with varying duplication of each:

Realm contains unsupported architecture:
AppleWatch app size exceeds permitted 50 MB
AppleWatch missing 44x44 icon
Framework8 contains a framework in its belly, and causes an error

The exact Realm error was:

Unsupported architectures. The executable for xxxx/Realm.framwork contains unsupported architectures [x86_64, i386]

Thankfully, I had just missed a step in integrating the framework…

As for the AppleWatch issues, I was easily able to reduce the size by compressing the icon images using ImageAlpha. How much, you say? saved about 7.5 MB!

The second, and more important time save was by setting the “Embedded content contains Swift” to “NO” in the AppleWatch app project settings. The reason I did this is because I noticed that 33 MB of the app size was due to swift libraries being embedded twice in the binary. So, setting the option to false on the main app reduced the size by 16.5 MB! Phew!

As for the missing icon, I had simply assumed that it was OK to only support Watch OS 2.0 icons, but I actually needed to add Watch OS 1.1 assets to the asset catalog, and accommodate that as well.

Finally, the embedded Framework error… Cocoapods strikes again! Framework8 utilizes cocoapods to link against SwiftJSON. In return, I integrate Framework8 manually using Xcode workspaces into the main project.. Apparently, you can’t do that for dynamic frameworks.

Invalid Bundle. The bundle at ‘X.app/Frameworks/X.framework’ contains disallowed file ‘Frameworks’

According to this issue, Cocoapods just insists on embedding the linked framework into its target even if the target is a framework itself. The suggested way to solve the issue is to convert the intermediate framework (Framework8) into a pod itself …

So, instead, I pull SwiftyJSON, build it locally, drop it into the workspace, add it to Framework8 framework search paths (without linking it!!), and finally make sure SwiftyJSON is added to the main app pod file. DONE.

Conclusion

Unfortunately, I didn’t write the issues I faced in detail, since I am writing this now after resolving all them issues .. Lesson learned for next time!

Shipping With Realm

2015-10-30T15:45:59+00:00

Introduction

I have recently announced starting to use Realm for my iOS projects. Then, I had just started using the technology, but didn’t really reach shipping stage until now. When it’s time to ship, that is when every ugly detail surfaces.

Note: This post is sponsored by fastlane! I found time to blog while it is busy building my app and generating screenshots.

Use Cases

For starters, I’ll be going over my use cases for using Realm, and then study each use case separately.

Shared Settings

The first use case is using Realm to store app settings (preferences), and sharing them across all bundles (Main app, extension, AppleWatch app). This was supremely handled well by Realm, and I probably can’t be happier with the setup.

Migration

Application settings change very easily and quickly. Adding more settings in the future won’t be a problem at all thanks to Realm’s clean Migration APIs. Breaking the settings down into small tables also helps a lot in managing migrations and changes, so keep that in mind.

For starters, all you really need to do is set the schemaVersion property on your Configuration object to 1.

Shared Container

First step is to set up an application group, and get a shared container going. The shared container will host the Realm files, and as per the docs, they can be accessed concurrently across processes without issues!

let fileManager = NSFileManager.defaultManager()
if let path = fileManager.containerURLForSecurityApplicationGroupIdentifier("group.identifier")?.path {
    // do something with path
}

Simple API

It is extremely important to have a simple API to read and update the settings, as well as extend the settings later. I probably have never had the pleasure of working with a cleaner API before, Realm takes the cake here as well.

To minimize duplicate code and stream line settings tables, though, I had to leverage Swift protocol extensions to roll out some convenient accessors.

First, let’s see how my current settings code looks like: Note: I use capital letters for static accessors

public class LocaleSettingsTable: Object {

    public dynamic var languageLocale = "en"
    public dynamic var regionLocale = "en"
    public dynamic var timeFormat = "12"
}

// get Locale settings:
let language = LocaleSettingsTable.CurrentSettings.languageLocale

// update Locale settings:
let newValue = "ar"
LocaleSettingsTable.CurrentSettings.commit { settings in
    settings.languageLocale = newValue
}

That is all there is to it! There is actually a lot of hidden power behind this concise syntax, so let’s dig deeper.

First, you should be asking: Where is CurrentSettings class variable coming from? Also, same thing goes to commit method?

Aha! Here is where Swift protocol extensions come into play. The common settings functionality is grouped into the protocol extension below:

public protocol SettingsTable: class {

    static var HostRealm: Realm { get }

    func preCommitHook()    // You can validate settings here
    func postCommitHook()   // You can apply more changes here
}

public extension SettingsTable where Self: Object {

    // NotificationCenter key for observers
    static var NotificationKey: String {
        return "\(className()).On.Update"
    }

    // lazy loaded settings
    public static var CurrentSettings: Self {

        if let settings = HostRealm.objects(self).first {
            return settings
        }

        let settings = self.init()

        try! HostRealm.write {
            HostRealm.add(settings)
        }

        return settings
    }

    public func commit(changes: ((Self) -> ())? = nil) {

        preCommitHook()

        let klass = self.dynamicType
        if let changes = changes {

            let realm = klass.HostRealm
            try! realm.write { changes(self) }
        }

        postCommitHook()

        NSNotificationCenter.defaultCenter()
            .postNotificationName(klass.NotificationKey, object: self)
    }

    func preCommitHook() {}
    func postCommitHook() {}
}

Now if you followed that properly, you’ll realize that any new settings table can easily get all these goodies by simply conforming to SettingsTable and implementing the HostRealm accessor.

extension LocaleSettingsTable: SettingsTable {

    public static var HostRealm: Realm {
        return Localez.SettingsRealm
    }
}

To wrap up this section about Simple API, I’ll just say that it was quite important to break my application settings into smaller parts and pieces. This allows Realm tables to evolve independently, therefor giving us granular control over each settings group.

Performance

Nothing to be said here about performance, really. We have a single object per table, and about 5 tables total. Everything ran fast enough for me to just ignore benchmarking this.

Further Work

To take this a step further, notifications should work across processes. I am not too worried or keen about taking care of that right now, so meh.

Key Value Store

The second use case for Realm was to be used as a key-value cache for downloads. The key type is String, however, the value is a JSON serialized NSData blob. Finally, we add an auto-updating NSDate() there to maintain the cache timestamp. This use case is also neatly covered by Realm + Swift generics.

Realm Table

If we take a look at the Realm part of this sub-system, it is summed up in a few lines of code:

class CacheTable: Object {

    dynamic var key = ""
    dynamic var timestamp = NSDate()
    dynamic var data = NSData()

    override class func primaryKey() -> String? {
        return "key"
    }
}

Generic Wrapper

As for Swift generics, it’s pretty lengthy an can’t be fully covered in this article. A quick skim, however, is in order. The wrapper is just two classes:

Task
Wraps the cache by providing an ability to instantiate a network task that automatically writes its result to the Realm cache.
TaskMeta
Facilitates the data that describes a Task. It is typically a URL which we sync the data from, a notification key which gets fired when the cache data is updated, and finally an expiration interval

Here is an example of how a Task is created:

var meta = TaskMeta(urlString: "\(hostname)/kpt/tweets")!
meta.expirationInterval = 1.hour.timeInterval
meta.notificationKey = On.TweetsUpdate

let task: Task<[AnyObject]> = Task(meta: meta)
task.updateIfNeeded()

You may notice: + Task<[AnyObject]>: This decides how the network response is parsed + 1.hour.timeInterval: This is just a convenience time library I use

Performance isn’t a big issue here. Since network operations are asynchronous by nature, we simply chain them with an asynchronous background access to the cache. Same thing goes to the deserialization.

Static Data

My final use case was to ship static data with my apps. I had two different types of static data:

Small number of “sounds” shipped with the app (39 total)
Large number of “prayer times” for all the cities in Kuwait (26k)

Manipulating Static Realm Files

Dealing with static data was a huge pain, in general, due to the lack of a interpreter interface. Every time I needed to manipulate the data, I had to write full-fledged swift apps to parse and manipulate realm files.

Once you do that, you’ll quickly realize it is no fun. Realm files are very rigid, and quickly throw errors when the schema changes, objects are copied between realms, and possibly the most annoying is dropping existing tables (not possible).

I swallowed my pride for a good of two days, and dished out two realm files. The sounds realm file worked to date without any real issues. However, the humongous prayer times file wasn’t snappy at all. Performance became a bottleneck.

Thankfully, my data was easy to break down into smaller parts, so that’s what I did. Each city’s prayer times was moved to a separate realm file, then there was a single realm file with all the cities. Once I did that, a realm file would have 2196 objects, and that brought the performance back up again.

API

As for the API, I was easily able to write a nice wrapper around Realm to make the queries really easy and simple. Here is an example of how the City objects are queried:

public extension City {

    private static func Results() -> RealmSwift.Results<CityTable> {
        return CitiesRealm.objects(CityTable)
    }

    private static func Results(cityId cid: String) -> RealmSwift.Results<CityTable> {
        return Results().filter("id == %@", cid)
    }

    public static func Query() -> [City] {

        return Results()
            .map { City(city: $0) }
    }

    ...
}

The main issue here is that I duplicate the CityTable class, and create a light-weight City struct that is returned to the caller. However, by doing that, I can now essentially switch out my core static data provider anytime I like without worrying about app wide changes. Also, I can control Realm’s complexity, since changing these Realm objects is a no-no (it’s a read-only realm!).

Conclusion

Realm is definitely better than any other persistence option, IMO. The only worry is performance. If you need that, you might as well roll out your own in memory map and use that.

Also, Fastlane has finished running a while ago, so time to get back to work!

Welcome to Realm

2015-09-09T00:01:28+00:00

Finally .. I’ve made the move to Realm.

What Happened?

Recently, I’ve been working an iOS project for some client, and obviously went with Swift for everything. The app is a very typical RESTful consumer, with a relatively simple data model.

For the first phase of the app, I focused on implementing and polishing the UI, and everything was great. Then, the time came .. I had to integrate with the backend.

I sticked to my guns, created a new core data model, and started creating all those various entities. Life was good.

While defining the models, generating NSManagedObject subclasses, and integrating the JSON mapping using MagicalRecord, things slowly started to break.

The first annoyance was when I got a crash from Swift’s unexpectedly unwrapped and found nil thingy. Apparently, generated Swift models don’t mark their properties as optional .. Not even implicitly unwrapped. I took a breath, downloaded Xcode 7 and generated the classes there.

Second annoyance was that now, everything was optional! Argh! Who cares, just wrap with some getters or use MVVM or something. Not a big deal …

Finally, two things happened simultaneously:

I was trying to get to-many relationship to work, and MagicalRecord failed.
Swift compiler crashed …

Seriously…

What now?

My best course of action was to move all the CoreData crap to Objective-C land, where I’m comfortable working with. Thinking of that made my head spin because of all the bridging management I had to do (nullability, generics).

It is a no-brainer. I shall use Realm, and it shall work.

Zero to One

I hope Peter doesn’t sue me on this one.. In any case, time is short, and lots of code to be migrated, so I had to start right away.

NSManagedObejct -> Object

Migrating the model classes to realm was a delight thanks to a few search and replace, regex-fu, and zmvs. The checklist is:

Change NSManagedObject superclass to Object
replace @NSManaged with dynamic

Inverse relationships were all removed, normal relationships were easily migrated, and the actual properties were assigned default values. Next!

NSManagedObjectContext -> Realm

Since I haven’t proven that this app needs background data saving and processing, I’ve been using the main thread for data operations. The app isn’t ready to ship yet, as well, so migration hasn’t been set up properly yet. I’m glad I haven’t invested in those yet, since look. I’ve completely rewired the data layer!

It was simple to migrate NSManagedObjectContext by creating a Realms.swift class with:

import Foundation
import RealmSwift

let MainRealm: Realm = {

    let config = Realm.Configuration.defaultConfiguration
    let realm = Realm(configuration: config, error: nil)!

    return realm
}()

I plan to add two more realms here, and setup all the configurations properly, but that’s about it.

JSON Mapping

Realm had recently published a talk that discusses using Mantle as a way to approach JSON serialization… I didn’t like that approach at all. As far as I could tell, it involved rewriting the model classes into Mantle subclasses, and .. so much duplication is what I was seeing.

Instead, I imported SwiftyJSON, and started rolling out my own simple mapper using convenience init on Realm Object extension. I bumped into a real annoyance here, but it was Swift’s issue:

The idea was that each model class would take a json, deserialize, then pass the json to it’s superclass, so it could continue building the object. That didn’t work.

Apparently, Swift doesn’t yet support overriding functions in Extensions (Swift 1.2). I mean, if you define a function in class A extension, and override it in class B extension, where B inherits from A, Swift compiler will complain. That annoyance made me break the model hierarchy into a flat hierarchy, where all models inherit from RealmSwift.Object.

That’s not too bad. The only annoyance is that I had to repeat the primary key declaration across all objects, instead of having a root class with an id property and primary key. Once Swift implements this feature, that code can go away!

import Foundation
import SwiftyJSON


extension ServerResult {

    convenience init?(json: JSON) {
        self.init()

        msg = json["msg"].stringValue
        status = json["status"].stringValue
    }
}

OK, I Lied

I really hope you are not reading this waiting for the part where I talk about hooking up Realm with your UI, since I didn’t do that yet ^^;. I have a rough idea how I will approach this, but it’s definitely not as neat as NSFetchedResultsController. Dat class is da bomb.

Conclusion

When the initial setup is just so smooth, and the issues I faced are just a tiny learning curve mixed with a few road-bumps unrelated to Realm, I gotta say my iOS development game is gonna change.

Realm is approaching v1.0 rapidly, with null-support just around the corner. I find it hard to think I’ll be switching back to CoreData .. Ever.

X-Men

2015-08-10T00:32:31+00:00

Introduction

This C++ language never ceases to amaze me .. Something new to learn everyday, and today I learned that mutating creatures aren’t X-Men exclusive, C++ has those, too!! The first time I was mutating was actually in Swift. In Swift, struct functions are const by default, unless declared mutating. For example, the code below doesn’t even compile:

struct X { 
    var y = 0 
    func mutateY() { 
        y += 1 
    } 
}

Anyway, mutating was found in C++, and it seems to serve a different purpose!

Injecting the Uranium

Let’s kick off with some code!

struct Nice {

   int sizeOfSharedContainer() const {
        Lock lock(_mutex);
        return _container.size();
   }

private:
    mutating Mutex _mutex;
    std::vector<int> _container;
};

Imagine the code Without the mutating keyword being there. Now, look at sizeOfSharedContainer implementation. You would have to assume that the Lock constructor takes a const& of _mutex, since the method is marked as const. Reality is, Lock needs to mutate _mutex, so other threads accessing _mutex know another thread is in the critical section! What now?!

The goal here is to still mark the method as const, since as far as the logical representation of Nice is concerned, the object didn’t change after the execution of sizeOfSharedContainer!! We mutated a mutex to ensure thread safety for a short while, then revert it back.

Lazy developers would immediately remove const from the method. Fortunately, the developer’s code I read wasn’t one of the lazy peeps. Adding mutating, as you probably guessed by now, allows us to overrule the const restriction for certain data members.

Why Haven’t I Head of It?

I guess I’ve never come across it before because it is essentially a double edged sword, much like const_cast. People use it very carefully, and it probably only makes sense once the program start to become a bit complex. Some workarounds exist, so this feature isn’t a dire need, so that’s also another reason why only few professionals use it.

Other Use Cases

Another helpful use case is lazy-loaded variables. Their getters should be marked as const, but on first access, we would have to mutate the cache variable to store the value once. Just make sure that cache variable is private!

A use case I actually thought of, and didn’t nab from the Google, is memoization. It is actually derived from the lazy loading concept, lol.

Conclusion

I need to write more interesting posts, with pics and stuff…

RAII

2015-08-02T22:38:51+00:00

Introduction

Unfortunately, we often overlook cool concepts in programming just because we simply weren’t lucky enough to have stumbled upon them during the course of our work. The blame is ultimately on us, for not searching for simpler solutions to our design problem .. I have no idea what kind of introduction this is, but it is one.

Initialization Is Allocation of Resource

Well, I spelled it backwards .. RAII stands for Resource Allocation Is Initialization. It also means, Resource Destruction Is Release. This principle, as far as I know, has only been widely used by C++ programmers, and after using it a few times, it simplifies resource management greatly, and fits amazingly well with the OOP mantra.

So, what in the name of mother bleep is RAII? Let’s just see a quick example:

{
    Lock lock;

    Data data;
    _vector.push_back(data);
}

What happened?

First, we initialized what looks like an “unused” lock object. Then, we create some data and append it to an instance variable … Doesn’t make much sense without knowing the internals of the Lock object.

The lock object is actually quite simply doing two things:

Acquiring a semaphore/lock on the code section in the constructor
Releasing the semaphore/lock in the destructor

If you think about it in the real world, you bring a lock, and now you have exclusive access to the critical section. after you finish your stuff, the scope will end, and the object destructor will be called automatically, so the lock will be released.

The great thing about this is:

It abstracts the details of how the lock is implemented
It makes perfect sense to OOP developers
It works well even if an exception was thrown in the middle!

Conclusion

Very simple concepts these days, but they are quite interesting for a mainly ObjC/Swift developer diving into the C++ world.

Multiplayer

2015-08-02T02:00:07+00:00

Introduction

I am really high on game development these days .. Much of my time and focus is on building this core game “framework” that will basically be the building block of all future awesomeness. I can’t make any compromises at this stage, as the technical debts accumulates insanely fast when you’re working without a large team.

Anyways, the idea is to either launch these amazing game series, or somehow get acquired for 500 million pesos. lol, yeah, I am high.

Abstraction

The main issue with designing a multiplayer game, from the client app perspective, is how to properly abstract away the network events, such that most of the app doesn’t really care where the events are coming from or going to. You don’t want to litter through out the code base with backend calls, obviously.

To deal with this design problem, we need to find the right abstraction. What does the network mean to our game? Well, for my game the network is just a pipe that sends local events to a remote player, as well as receive events from the other end of the pipe … So, the abstraction unit for the game is the player!

Indeed, imagine you are sitting in a checkers game, and you are playing against an AI, like physically. The game board is the same, you are you obviously, but what changes is that instead of sitting in front of a human being, you will be sitting in front of a robot or something. Same thing goes for a remote player.

Materialize

Now, we need to materialize the abstraction, and actually implement this setup.

By creating an abstract player class, we can have a common interface for the main runloop to deal with the players without caring what type of players they are. Basic OOP stuff, moving right along, since designing the player interface is what is actually interesting.

To do that, let’s see what information a real physical player receives throughout the game. A player would obviously be looking at the game board, and knows the game state, so we need to pass the GameLogic to the player. The player also would see their opponent making moves, so we either send the move to the opponent, or simply pass a GameLogic update. That way, the player would also know its their turn now.

That is pretty much it. Anything else you add to the player might be redundant or a mistake in the abstraction. When I first designed the game, I thought the player owned her pieces, but that just complicated things, and it was better to just keep the pieces as part of the game board.

Conclusion

I really hope this post was actually useful, since I didn’t even bother add code snippets nor images, so it looks quite dull … Anyways, happy coding!

Graphic Agnostic

2015-07-29T05:33:44+00:00

Introduction

I’ve recently committed to writing my whole game on terminal. This had two great advantages:

1. Worry about graphics later I am 100% focused on the technicality of the game, and low level experience. I don’t have to worry about graphic details just yet.

2. Maintainability Using this approach separates the concerns pretty well. This helps us achieve maximum maintainability and testability. UI testing is a pain since the middle ages, and what better way to solve the problem than remove the UI altogether?

The Thing

I don’t really have the time nor motivation to write down all the details, but I made a quick video tutorial about the subject. You can see the project structure, and terminal client, and how awesome it is to develop with this approach:

Youtube link

Conclusion

The main thing to keep in mind here, do whatever works for you. I am a real freak when it comes to details, and I want to perfect the app and experience in every way possible, especially technically. That is simply not possible without a solid code design, so having that is extremely important for me.

Will You Be My Friend?

2015-07-23T23:37:28+00:00

Introduction

A friend is someone who you can share private secrets with, and they will never betray that trust. A friend is someone you choose, not the other way around. A friend makes your life safe by helping you out.

In short .. Friends are awesome.

Friends in C++

In C++, we have things called “friend classes”. I remember our TA at college making fun of us when we answered the trick question “Can private members ever be exposed to an outside class?” wrongly..

Ehm, anyways. As mentioned in the introduction, a friend class is a class you share your private members with. You choose which classes are your friends by declaring them as friend classes in the class definition. Finally, as we will see, this feature will allow you to safely control access to specific “dangerous” members.

The best thing about this feature is the syntax. The word “friend” is just so expressive, and to use it is a simple one liner:

struct Block {

    Block(short row, short col);

private:

    Block() {};

    short _row;
    short _col;

    friend class TheFriendClass;
};

Use Case

After writing Swift code, and dealing with optionals, I really fell in love with this concept (Optionals). It eliminated null pointers and empty value issues in one go. Now that I am working on a C++ project; luckily, I found that POCO has Optionals (and Nullables)! I started using this feature heavily right away.

The biggest issue that I faced was that, in order to use Poco::Nullable, the type had to have the default, no-argument constructor. This is because the Nullable class had to always initialize the type memory to something, even if it was null. Of course, pointers weren’t used, because if you want dynamic allocation, just go use std::shared_ptr instead.

What if the type you want to use with Nullable doesn’t provide a default, no-argument constructor because it doesn’t make sense to have one? Yeah, it should be obvious now … Friend dat class!

By looking at the example above, and replacing TheFriendClass with Poco::Nullable<Block>, we safely expose the unreliable no-argument constructor only to the Nullable class, so it can just initialize that memory to something. So, we are 100% certain now, that Block objects can’t be initialized with the default constructor, unless they are safely within the Nullable class, which won’t operate on the instantiated instance, anyway!

Conclusion

Nullable implementation wouldn’t be possible without generics, and thankfully, C++ had that feature way back. I still get developers asking, “Are generics really useful?”. Yes. Yes they are.

Basic C Programming

2015-07-22T13:14:11+00:00

Introduction

After all this absence from my blog .. I just have to add an introduction! Well, since I stopped blogging a few months back, I was actually back home and for some reason, life is so easy going there, that I end up not able to work as effectively.. Weird. Anyways, my travels have resumed now, and life is tougher than ever, yet I now have time to blog! :D

Now, for an introduction about the actual topic…

C programming isn’t something every programmer stumbles upon anymore. Even though I am mainly an iOS developer, and have written tons of, technically speaking, C code mushed together with the ObjC code, I never realized the “proper” was to write a well structured C program until very recently.

Vitamin C Deficiency

One of the first things I miss when coming to the C world are classes. After learning and writing C++, ObjC, Python, and especially Java, classes are just this central part of my design. I dealt with top-level functions in some of those languages, but in very limited, and usually hackish ways.

The lack of classes, meant lack of methods. Methods are insanely useful. Imagine not being able to write constructors for your classes. That will lead to so much boiler plate as you initialize all those properties whenever you instantiate a new instance of a class.

Of course, the language lacks a lot of other useful features, like closures, generics, exceptions, .. etc, but today’s focus is on the classes.

Taking Supplements

For some reason, it never really clicked in my head till I recently started reading some well written C libraries .. Lack of classes is not that big of an issues, especially where methods are concerned. To emphasize on this point, take a look at go by Google. It doesn’t have methods the way we are used to them in OOP languages.

So, the realization was, and I feel so stupid for realizing this just now, is to simply do the following:

Imagine you have a C++ class with a method
Take that method to an outer scope
Append the class name to the beginning, to add more context
add a first parameter, which takes a pointer to an instance of that class
Done!

As you can see, we are simply doing some more manual work to what the compiler used to do for us automatically. When we call a method in C++, we can use this, which is but the instance the method is called on, passed to the method automatically by the compiler. And, instead of using the nice . syntax, we simple have a function prefixed with the class name.

Here is a sample from the library I am working on:

typedef struct _DMGameMode
{
    eGameType gameType;
    eColor color;
    eTimeLimit timeLimit;

} DMGameMode;

extern DMGameMode DMGameModeMake(eGameType);
extern DMGameMode DMGameModeMakeSingle(int level, eColor color);

extern int DMGameModeDifficulty(DMGameMode *mode);
extern int DMGameModeMaxDepth(DMGameMode *mode);

I add DM prefix for the lack of namespaces. Then, we have constructor like functions, and later methods that operator on an instance of that struct.

Conclusion

I knew that, for example, in ObjC method calls are translated to objc_msgSend calls, and self, _cmd are passed in, but it didn’t click that this is how developers generally write C code. I was mostly concerned about using pointers, but it’s not that bad after all.

'HTTP Game Server: The Rewrite'

2015-05-05T18:03:22+00:00

Introduction

I’ve stopped blogging for a while, and for no good reason, really. I will not make up any excuses, but will try to get the ball going once more.

The Past

I’ve been blogging recently about my experience regarding building a game server on top of Google Appengine, and it was a nice journey. The game server was running pretty nicely, the tests were well integrated, and everything seemed to be on the right track. Except, when you think everything is going fine, nothing is.

As of last month, I completely steered away from the Google AppEngine platform, and no longer maintaining that HTTP game server. I am glad (I think) the game hasn’t been shipped yet, so I can easily change gears without really affecting anything.

But, what happened?

As I started writing more and more client code, things quickly started getting ugly. In order to fix the client, I needed to update the backend. The backend already had tons of tests, so I needed to update those as well … That’s when I lost the last thread of patience with Google AppEngine.

To be honest, the issues isn’t really with Google AppEngine in itself, rather it was the language and the Restful API design.

What Now?

In order to write proper client networking code, I started utilizing the Poco C++ libraries. After using those libraries and tinkering around with other library components, I realized how amazing this library is, and how I can simply write the whole goddamn system on top of Poco, so it started.

So, halfway through writing the client networking library code, I split the common networking part into a separate library, and started a new C++ server project that utilizes this shared library. Now, changing a single library will update both client and server!! NEAT!

Not only that, but I also lost a shitload of time trying to implement a restful service in C++. After going through the implementation more and more, I kept refactoring till I finally reached a very minimal and functional client-server architecture!

I reached this amazing design about 3 days ago, and I already have user registration, login, sessions, and matchmaking up and running. I had few small issues, but progress has never been faster. Finally, I can say that all that is left is grunt work of rolling out all the functionality, and then integrating it into the game.

More Guts and Gore

To share more insight on the design, here are some details:

What is Shared, Exactly?

The shared code is mostly entity classes that know how to serialize and unserialize themselves. This allows us to instantiate objects on the client/server then transport them and recreate them easily on the other side. Since it is a shared library, changing things will automatically reflect on both sides.

This means we don’t even require a transport spec between the client and server, since the serialization happens in the shared library as well.

Another amazing benefit is sharing actions and results. Now, whenever we want to perform an action from the client, the actions are all defined in the shared library, and so are the possible results. This allows the server to know what actions are possible, and the client to know what the possible results are, as well. Amazing.

How to Write Such Things?

This is C++, and it is not the most usable language. Actually, it is by far the most unusable language I care about. I had issues with templating, passing data around, but mostly, objects life time.

After learning about the benefit of value types, and how they contribute to reducing overall system complexity, I dedicated myself to write as much value types as possible. I don’t really care at this point about efficiency, I care about correctness. There are some edge cases where pointers and dynamically allocated memory was necessary, and std::shared_ptr was the star there, and I never had to manually manage any memory at all.

To top it all off, I dedicated myself to writing a single threaded server. There was no need to deal with multithreading and the dreaded consequences that comes with it. Instead, even though the server is multithreaded, it dispatches all the work on a queue that gets consumed by a single thread. All the business logic runs exclusively on that thread.

Conclusion

Hopefully this will lead to a HTTP Game Server series rewrite, and if you are lucky, I really want to open source this and make it a starting point for all game developers out there.

Benefits of Unit Tests

2015-04-13T10:53:37+00:00

Introduction

There is a very common problems among programmers that haven’t tried unit tests .. They don’t know how to start. Most of the programmers that I’ve met, including myself, had this problem, and the natural way to go about this is to buy a book and start reading.

Problem is, I had some really bad experience with books on that topic, so did the people I met.

Today, I have finally had enough unit test experience that allows me to explain how to go about this.

Clarity

Let’s be clear first … There are many types of testing, unit testing, system testing, integration testing, black-box testing, white-box testing, … etc. Make sure you read about all types of testing methodologies, because they will teach you what unit tests aren’t.

To highlight some of the important topics here:

Unit testing isn’t BDD

BDD is writing high level tests that the business understands and can be expressed in terms of the system output. That means you don’t really care about the internal workings of the system as long as the correct output comes out.

Integration tests, functional tests, system testing aren’t much different. They only differ in subtle ways, but they are all high level testing that usually doesn’t need knowledge of the internal system workings.

What Unit Tests are

Unit tests are tests that are written for every single part of your system. You test that your object contains certain properties, you test that invoking a certain method returns a certain result, and you even test the output signals generating from your system. It is a very low level rigorous testing.

Learnings

Now, here are the enlightenments I had while writing those tests…

No Duplication

The fact that you have to test every single operation in your system makes you never want to write duplicate code .. ever. If you write duplicate code, you will duplicate your tests, and things will start looking shitty. I have a rule that, once you start copy pasting your unit tests, that means your program isn’t DRY enough, or you’re doing it all wrong.

One reason you might end up writing duplicate tests is not necessarily duplicate code. You might be mixing BDD with TDD. Here is an example:

class Component:
  def __init__(self, operation):
    self.operation = operation

  def perform(self, input):
    return self.operation(input)

Notice that the component receives an operation that changes the behavior of this Component class. How are you suppose to test this?

If you are new to TDD, you might be tempted to instantiate a Component, set the operation to something, and then make sure that operation performs properly. That is not proper TDD, because you will be duplicating the testing of the operation, which is external to this class!

The test for this case is to simply pass a dummy operation with a mock object, and assert that when we call the perform method, the mock object will be called:

class Stub:
  def __init__(self):
    self.did_call = false
  def __call__(self):
    self.did_call = true

Component(Stub()).perform()

I hope the above example is clear. We are simply testing that if we pass an operation to the Component, the Component will perform a call using that operation when perform is called.

Fortunately, you are a passionate programmer, you can’t leave things the way they are! So, while writing unit tests for the system, that will be a perfect opportunity to find and eliminate duplication, and discover better ways to come up with abstractions.

Break Dependencies

Imagine if for you to write a test for a function that is the handler of a button click. So, on button clicked, you want to send an HTTP request, and schedule a notification (those are totally random things, but you might be doing something similar). Testing this function because almost impossible.

You have to instantiate the HttpClient singleton, initialize the notification scheduler somehow in order to for the function to be tested properly .. That won’t do at all.

The rule is:

When writing unit tests, you must test each component in isolation.

Taking the previous example into consideration, we first need to make this function serve one purpose, and then test that alone. So, what we might have is another core component, completely separated from the UI, that receives a signal through some NotificationCenter.

This way, all we have to do to test this function is make sure that it submits that notification! As for the other stuff that needs to happen, we can design it in the observing component.

In this case, the component might have services registered to handle certain signals. So, there is an HTTPService that registers for that signal, and will send a request when that signal is received. Notice, that now we can test this HTTPService separately! The test is to make sure that when a signal is generated, the HTTPService attempts to send an HTTP request.

Catch Bug Early

Needless to say, that one of the toughest things we programmers go through is finding the origin of a bug, especially a semantic one. By unit testing, you can verify all your assumptions you are making by simply running the tests. That will help you catch bugs faster, because you are testing your changes at a micro level, before running them at a macro level.

This is especially useful when you are writing C/C++ code, or in some scripting language, since it’s easy to make mistakes there. Something could easily slip by because of the extreme low-level/high-level programming.

Conclusion

Yeah, I am not gonna go through other obvious TDD benefits that people evangelize in books, since this post is all about practical advice that I actually experienced.

POCO

2015-04-09T09:21:51+00:00

Introduction

Recently, I’ve quit my job, and started doing lots of things I had in mind for years now. Point is, I am my own boss, and I would like to make stuff as perfect as possible.

When you are working in a very small team (1) the number one priority for you becomes speed. It doesn’t matter anymore how robust your system is, how performant, how scalable … etc, as long as you can get stuff done quickly and ship! Otherwise, welcome to the endless world of programming indulgence.

With that said, you ultimately have requirements, which are the absolute rules you can never violate. You may be able to bend some, but some hard requirements just have to be accommodated.

For my stuff, it has increasingly became apparent that I need to scale. I have verified the success of a few projects on the iOS platform in the local market, and now it’s either expanding internationally or to other platforms .. Hopefully, both!

The number one language for mobile portability is without doubt C++. Apple uses C++ internally for loads of frameworks, so does Dropbox in their mobile app to share code, let’s not forget Realm, it basically C++ core, and last but not least .. Facebook’s fresh ComponentsKit. The C++ there actually goes beyond the internals, and is exposed in the interface!

Bottom line, brace yourself for C++ programming if you plan on going cross platform. Good news, though. We’ll see why it isn’t so bad to write C++ code anymore.

What Was Wrong?

Back in the dark ages of pre-C++11 (C++x0), C++ was such a painful language with very limited features that made it very lacking when compared to newly created modern languages.

I entered the C++ world by working at a game during my time at Sourcebits. The team, (which I hope they aren’t reading this now) including me, were “very behind” when it came to pure C++ programming. I mean, it was 2013, and the team didn’t know about the foreach syntax, lambdas, and other essential productivity features. These features were replaced by ruthless labor of basic design patterns. The code was horrible.

I can’t really go into much details here, because to me, C++ was so horrible, I might’ve quit programming if it was the only choice. Hence, never programmed much in it during that time, as I sticked to working on tools and scripting with ObjC and Python.

That all changed, because it had to.

What Changed?

After that experience, I was a bit more ready to use C++, since I’ve learned after that job about how C++11 changed everything. Downloading Cocos2d-X and learning from it’s opensource galore helped a lot in utilizing the new features.

After about a year or so doing that, I wasn’t going anywhere. Dealing with Pointers, memory management, and other awefulness imposed by the Cocos2d-X engine for the sake of performance was really getting to me .. Remember the first thing I said in the introduction? Speed can’t be compromised!

Meanwhile, Swift came out, and it taught me an insanely important lesson. Don’t fool yourself. Language matters, and productivity is greatly effected by choosing it, so choose carefully. Also, make sure it’s the language, and not that you are just a bad programmer.

So, what happened was I realized I was a really bad C++ programmer. I approached C++ the same way I would approach ObjC or python, which was a huge source of pain. Classes where unnecessarily humongous, every class depended on every other class in the system, and yeah. I didn’t utilize any library.. !

The New C++

The new C++ to me is POCO.

By seeing POCO’s modular system and insane set of features, everything clicked. C++ is a great language, just need to utilize all available resources!

It is very important to break every single component into small pieces. Attempt to make it generic, as well, so it doesn’t have many dependencies. Finally, utilize the hell out of the POCO library itself! The feature set it has and cross platform compatibility/portability is worthy of high praise.

But…

I know, still, this is no silver bullet. Integrating C++ with other platform languages in order to leverage native frameworks is usually painful. NDK on android isn’t the most usable thing, nor C#’s dynamic plugin loader, but by investing the time upfront to put a solid build system, it could really payoff.

Tools like Djinni, SWIG, … etc help, probably, though I have yet to start using them.

Conclusion

I’ll leave with this statement as food for thought:

Premature optimization in C++ is worst than being a really bad programmer.

Enlightenment

2015-04-05T22:17:27+00:00

Introduction

I’ve been blogging recently about Swift and SpriteKit, and that’s mostly eating up all my time, making me miss a blogging day, and also writing a completely useless post.

Now, to justify this, the previous three posts before that have been quite lengthy and informative, so it’s only natural that the bigger you are the harder you fall, lol.

This post is just me trying to recover from the fail streak I’ve been having.

What Happened?

Actually, what didn’t happen is I haven’t shipped my game yet. After three years of development on the side, it should have been released ages ago, and even new versions already out, but no … That never happened.

What did happen after that is I built a solid game prototype with my partner in two weeks. Two weeks vs Three years. And I am not talking about a throw away prototype, here. My partner suggested a throw away prototype, but I just went ahead and made it as solid as possible, with a simple yet rich architecture, that can easily evolve in the future.

How Is That Possible?

The infinitely long game was developed in Cocos2d-x using C++. It was initially built by a dump port of Objective-C code, and sustained all cocos2d-x’s updates … It evolved into a really shitty beast. Adding new assets, moving things around, changing animations speeds, and all that still takes me a few days to get right, that is insane!

Meanwhile, the new game prototype uses external tested components, and a very small footprint core engine that drives the game. Making new builds and testing takes seconds. And of course, the language I am using is Swift. It is truly a reflection of its name.

What Now?

Now, it’s time to upgrade into a real game developer.

Game development in itself is such a daunting and difficult task and doing it part time with hacks and workarounds isn’t sustainable. I did that a lot in the app development business, and it worked quite well, especially with a development cycle less than 1 month. Again, not in game development.

It’s time to actually architect each component separately, test it, and decide on the integrations. You’d probably be reading this and saying “But, of course!”. Well, I am the only programmer in the team, and buying in to such approach is a huge time investment, that requires significant solo experience, since you don’t have much people around to bounce ideas off of.

Conclusion

In a nutshell, if you are doing game development for real, and looking to have that as you long term goal, STOP hacking and START building!, and remember, it’s not a sprint, it’s a marathon.

Brain.Dump()

2015-04-02T21:21:43+00:00

So much freaking work to do, so little time
Awake for more than 28 hours, and counting
Game development in Swift is just amazing
SpriteKit is crap, and crashes unpredictably
We have an amazing prototype already! Yay!
The component system built is a piece of art
I must blog or release the source someday
Hopefully I can finish this last feature
I think my brain is giving me 4 more hours
Writing code in such state is amazing
I see stars
There is a cup of coffee fist bumping his friend
What day is it today?
I think I drank 4 cups of coffee today
You know that feeling that you’re missing something
Well, I feel I missed a meeting with the emperor
Nothing makes sense anymore, but I did post!
HabitRPG character will be pleased
So will my wolf
Princess Monanoke was an amazing movie
It reminds me of Kiki’s delivery service
Kiki is my X-roommate’s nickname, too
He recently went to Bahrain
Bahrain means “Two seas” in Arabic
Only way to have to seas, is by accelerating
ha! I am not funny
This hole thing was just a waste of your time, sorry
no refunds available
Hopefully you now realize how important time is
Wow, I actually turned this into something useful
I must stop lying to myself
This will be the best SEO post of all time
I wonder if I should stop typing ..

Marry CircleCI to Hockey

2015-03-26T15:10:25+00:00

Introduction

If I were to tell you, today we shall marry CircleCI to HockeyApp, who do you reckon would be the bride, and who would be the groom? Doesn’t really matter actually, as nowadays, that’s not the only allowed typed of marriages!

Enough with the silly introduction, and onto the l00t.

Purpose

CircleCI is an amazing service that provides continuous integration for various projects, including iOS projects.

Continuous integration in its “final boss form” is a service that checkout the latest code, performs some automated changes and formatting, builds the product, runs automated tests on the product, performs post testing actions, and finally deploys the product.

In a typical iOS project, the deployment phase is not necessarily deployment to the AppStore directly, but rather deploying to a distribution platform which can provide the latest builds to your QA team. For all we care, your QA team could be 🐵🐵🐵 that tap all over the screen and get bananas as a reward.

Now, it could easily become a chore if you had to wait for the CI process to complete, before deploying to the distribution platform, hence the purpose of the article. Let CircleCI take care of that distribution.

Code Signing, Provisioning, and Archiving. Oh, My!

First of all, take a look at this superb example of how things should work. The sample project shows how you can distribute TestFlight builds through CircleCI, so with a bit of tweaking and explaining on my end, you should be all set with minimal head-scratching.

Ground Work

Before doing anything else, if you want to commit to making builds on a server, you need to make sure the following checklist is accommodated:

Device UUIDs registered
App ID for the product
Ad Hoc Distribution Provisioning profile
iOS Distribution certificate

Once you’ve created those in provisioning portal, you’re ready to move to next step.

Exporting

Now, we need to have the required files in one place, and we will use them in a second when we configure the repo. These files can be exported from your keychain on OSX:

Apple Worldwide Developer Relations Certification Authority You can search for it in your login keychain, and just export a cer file, call it apple.cer.
iPhone Distribution certificate Your iOS distribution certificate exported as a cer file, call it dist.cer.
iPhone Distribution private key Your iOS distribution certificate exported as a p12 file, call it dist.p12.

NOTE:

When you export the private key, it will ask you for a password. Take not of that password, we will refer to it later.

Now, either download your mobile provisioning profile file from the portal, or you can find it located at the following path:

~/Library/MobileDevice/Provisioning Profiles

I use the excellent quick look plugin to inspect the files.

Anyway, once you have your hands on the profile, copy it “as is” to where ever you put the previous files.

Setting Up the Repo

Now we are ready to configure the git repository that will be checked out by circle ci.

First, let me be clear:

Here is where I made changes to what the github example has

In the github example, the developer prefers to run a bunch of custom scripts in order to generate an archive, process it, and upload it to test flight. In my case, I decided to simplify these steps as much as possible.

So, with that out of the way, let’s see how can we make this happen:

Xcode

Some changes need to be made to the project itself, first. There must be a build configuration that has the code signing and provisioning profile set manually. In my case, I created a new build configuration, and set up the code signing and previously created provisioning profile:

You can use Release if you like, or call it Distribution, just make sure you take note of which build configuration is using the distribution identity.

This means we don’t need the testflight.sh script in the github repo.

Environment

In order to upload your build to hockey, or any other service really, you need an access token. In order to secure the token and conveniently change it through UI, we will set it up through Circle CI project settings.

Remember the exported private key p12 file that you password protected? Remember how I said to take note of the password? Here is where we will use that password, as well.

Just go over to the project settings on circle CI, and set up the environment variables with HOCKEYAPP_TOKEN and KEY_PASSWORD. The hockeyapp token can be generated from the user account settings -> API Tokens.

Here is how it should look like (you don’t need all of those):

Scripts

The scripts we will be using can be grabbed from the aforementioned github repo. We only need the add-key.sh and remove-key.sh scripts. Just grab them for now, and we will see where they go in a bit.

Circle.yml

circle.yml is a configuration file for circle CI that we need to setup. If you don’t already have that file, please create it at the root of your repository.

Now, for the easiest possible solution, I use Shenzhen for build and distribution. Notice the -c Distribution flag is to choose the build configuration we already setup.

You are free to use xcodebuild, xctool, or whatever you wish, but if you do that and still want to leverage Shenzhen distribution, you must also use xcrun to get an ipa file. I will not cover that part, as I will be using shenzhen for everything.

Shenzhen will need a hockeyapp token, and notice we will be using the environment variable we already set in the circle CI UI previously.

dependencies:
  pre:
    - sudo gem install shenzhen
test:
  override:
    - ./scripts/add-key.sh
    - ipa build -c Distribution
    - ./scripts/remove-key.sh
deployment:
  hockey:
    branch: master
    commands:
      - ipa distribute:hockeyapp -m "circleci build" --token $HOCKEYAPP_TOKEN

The Files

Now, copy all the files we covered in this post so you have the following hierarchy:

ROOT
- circle.yml
- scripts
  - dist.cer
  - dist.p12
  - apple.cer
  - add-key.sh
  - remove-key.sh
  - profile
    - blah.mobileprovision

Bringing It All Together

Takes a Deep Breath

In a Nutshell

So, we created all the certs, profiles, app ids, and configured the xcode project with those by adding a new build configuration. Then, dropped those files along some scripts into the repo, configured the circle.yml to generate a signed build, and uploaded the build using shenzhen.

Conclusion

Frankly, using shenzhen is a bit cumbersome since it has tons of dependencies on different distribution platforms, but it is just so darn simple. I will eventually move away from it, but for now it works well.

There might be minor issues regarding ASCII stuff, just define the LC_CTYPE to utf-8 or something, unfortunately I don’t remember why I did that part.

With all that done, you should be able to upload test through circle ci!