When working in Node.js, you're never working on a project alone. The runtime, the package manager, and tons of the common libraries are all the work of thousands of developers around the world and they are available to you right now.
Learning to leverage the javascript ecosystem is important to being a productive node.js developer. In this article we will explore the process of creating a program using some of the libraries out there. We will do it while producing value for a small business owner named Frank.
Understanding Frank's Needs
Frank needs some help. You see, he runs a small gym and there is never enough time to get everything done. He works days training people in his gym and nights managing the couple of other trainers he has hired. Frank doesn't have a lot of time left over after working out the day's clients, fielding a call from a prospective new client, cleaning up his gym, balancing the books, and all the other things that have to get done.
Despite his lack of time, Frank would really like to get some feedback from his clients about how they are feeling after some milestones being trained at his gym. Every week Frank calls anyone who joined in the last week to ask them a question or two. The client has only been in Frank's gym a couple of times and he wants to make sure they feel comfortable and that they feel good about the next week. After a month Frank makes the same call to judge his progress with the customer. As his business grows, and he hires more trainers, the number of clients he has to call is getting out of control and so Frank has reached out to you to for help getting feedback from his clients without requiring so much of his time.
After a couple of talks with Frank, and some of his new clients, we discover that they are open to receiving text messages rather than phone calls and that there are only a couple of key questions that need to be asked each time. Now that we know the needs of our client, Frank, we can begin to think about what software to create in order to solve the challenge facing his business.
Determine a solution for Frank
We learned a few things from interviewing Frank. We learned that he needs to get in touch with people using the phones they keep in their pockets all day, and we learned he needs to ask a couple of questions, record their answers, and track the results over time.
Twilio is a service provider that gives developers the ability to put a program behind a phone number in the same way that they've been able to do with URLs for over 20 years. Twilio provides an API for sending and receiving SMS, MMS, VoIP and traditional voice calls.
Node.js is very well equipped to respond to events (for example, a client's first week milestone passing or an incoming SMS) has an official twilio library and has a ton of other libraries available for interacting with databases, crunching data, producing graphs/reports, and building web interfaces (for configuring the sms's text and frequency).
In order to begin helping Frank ASAP we need to ship quickly. To facilitate this we will start with the minimal viable product: a script to send a first week, first month, and third month survey via sms and record responses in a database. That will give us ample time to build a more robust solution. It will also give Frank time to determine what pains him most about this basic solution.
Note: I've written about some of the common javascript idioms you'll need to be familiar in order to work with the Twilio library elsewhere on this blog
Build it
For the purposes of building this tool we will make no attempts to future proof the application. That will just slow us down. Instead, we will take only the requirements as they exist and create a solution that works, not one that tickles our engineering sides' fancy . Don't worry though, we'll build a gorgeous version once we have the full set of requirements.
For this solution we will use mongodb as our database because Frank's existing client data is already stored in a mongo database. We will use the high level mongodb node driver to connect and query the database. Additionally, we will use Twilio to send and receive SMS via the official Twilio node.js client library.
With the exception of the mentioned dependencies, we will only use vanilla Node.js 5.0.0 stable and the bundled version of NPM.
In order to do development on your workstation, it is a good idea to have a local instance of mongodb running. Please see installing MongoDB for more information.
To use Twilio to send SMS using this application, you'll need an SID, AUTH_TOKEN and PHONE_NUMBER from Twilio. You'll need an account to get started. You can get one here. Please refer to their documentation for the process of obtaining the necessary data.
Note: For those of you looking for mentions of gulp/grunt, I hate to disappoint. I'm Keeping It Simple Stupid for now in order to focus on the core application. We will get to setting up a task runner and build automation soon enough.
Getting Started: Packaging
With a Node.js project, it's often a good idea to start by running npm init which will guide you through the creation of a package.json file. package.json is used to describe a package including its dependencies, name, version, and information about its author. For this project it'll start looking like this:
Since we know which dependencies we will need for this project, now is a great time to install them.
npm install --save mongodb twilio
The --save flag will indicate that the installed packages should be recorded in package.json, which now looks like this:
Notice that there is now a key called dependencies which contains information about the other packages this one depends on. By bundling the dependencies' information with the package.json file, you don't have to redistribute them with your app (which will save you a ton of bandwidth and other headaches).
Ramping up: The Database
Now that we have a package defined and the mongodb library installed, we can get connected to the database. Because the code for connecting to the database is a single, isolated bit of functionality it makes sense to break it off into a module. Documented version
The database module's primary job is to provide a simple mechanism for getting a database connection pool handle. In doing so, it deals with the potentiality of an error while connecting to the database as well as define a default database url after attempting to read one from the environment.
Client Schema
Having a connection the database is only a part of the story. We also need to have an understanding of the structure of the data. For Frank's gym there is only a single data model, Client. It is defined as:
Every field except for name is essential to the success of this program.
signupDate is how we determine if the client has existed for long enough to meet a milestone.
phoneNumber is the information necessary to reach the client by sms
responses is a record of the messages received from the client. This is how Frank will review the replies from his clients.
wasSentSMS is a collection of flags that indicate which milestone SMSs the client has already received and prevents us from spamming them.
Functionality
The schema of the model describes the structure of the data. In order to control the manipulation the data, there are also some functions that get attached to the objects. These functions serve to group database changes into a complete interaction so that there aren't bits of code all around that manipulate the database, which eliminates a whole class of bugs. Those functions that augment the client model can be found in context here and appear below as a snippet
Fixtures
When we are building the application, we won't always be connected to Frank's database. In fact, while testing things out is hardly the time to be playing around with Frank's client data! Fixtures make it easy to test the application locally by inserting example data into the database. In this case we are simply inserting some POJOs into the database by converting them into Client objects and using the save method.
Fixtures are also useful (as opposed to inserting documents arbitrarily into the database) because you can easily repeat the insert and the data will always be the same on every new insert, it will be easy to throw out the data when you want to reset it.
Hello, Twilio!
After successfully gathering the tokens necessary to use Twilio, you'll need to set those values. Because it is bad practice to commit sensitive information (like Twilio tokens which can be used to take over your phone number) into source control, it's best to read those things from the environment in which the node application is running. In Node.js that is done using process.env like this:
Having retrieved the configuration values from the environment, building an sms module becomes quick work. Its purpose is mostly to deal with configuring Twilio and separating the details of the SMS service provider from the business logic of needing to send an SMS.
Note: The module, in its full glory,as it appears on Github, doesn't require Twilio to be configured to crank up. This is entirely for convenience when working through the source code without setting up a Twilio account.
Incoming!
In addition to sending messages, we also need to record incoming SMS from clients. The first part of this, retrieving messages, is made pretty simple by the Twilio client library. The trickiest part of it is ignoring messages from strangers because our data isn't already stored in a manner that is conducive to looking that up easily. Luckily, it is fairly straight forward to turn an array of objects into an array of values using Array.prototype.map.
The second part involves storing the incoming messages with the appropriate client. The important part is to de-duplicate the responses before storing them. Because every call to SMS.client.messages.list will return every message ever sent to the specified twilio number, we will potentially store a message multiple times if we do not ensure that we ignore messages we've already stored.
Start 'er up!
Now that we have all the modules we will need to do the work, it is time to write the business logic. This code is the most tied to Frank's gym of everything else that was written. It is responsible for looking at all the clients, determining who needs to receive messages, and sending those messages.
The main loop is a setInterval that will reset itself if it encounters an error. At boot, and once per interval (1 day in this case), the program will grab all the clients from the database and consider sending them each a survey. It will also check for any new incoming messages from clients and store them.
Note: While we haven't reviewed every line of the application, we have covered which pieces are necessary and how they interact. You can view all of the application's source code on Github. It is thoroughly documented and should provide a higher level of insight into each module.
Ahem!
Frank is happy, the first wave of messages are sent and new ones will go out every day, but our job isn't done yet. There is a lot that can be done to make sure that this important part of Frank's business operates smoothly.
Tune in next time when we write tests for this application so we can sleep well knowing that the application will hum along as we expect it to.
... Speaking of tests, which tools would you like to see used? Jasmine? Mocha?