1. Code
  2. Go

Testing Data-Intensive Code With Go, Part 2

Scroll to top
Read Time: 9 min
This post is part of a series called Testing Data-Intensive Code with Go.
Testing Data-Intensive Code With Go, Part 1
Testing Data-Intensive Code With Go, Part 3


This is part two out of five in a tutorial series on testing data-intensive code. In part one, I covered the design of an abstract data layer that enables proper testing, how to handle errors in the data layer, how to mock data access code, and how to test against an abstract data layer. In this tutorial, I'll go over testing against a real in-memory data layer based on the popular SQLite. 

Testing Against an In-Memory Data Store

Testing against an abstract data layer is great for some use cases where you need a lot of precision, you understand exactly what calls the code under test is going to make against the data layer, and you're OK with preparing the mock responses.

Sometimes, it's not that easy. The series of calls to the data layer may be difficult to figure, or it takes a lot of effort to prepare proper canned responses that are valid. In these cases, you may need to work against an in-memory data store. 

The benefits of an in-memory data store are:

  • It's very fast. 
  • You work against an actual data store.
  • You can often populate it from scratch using files or code.

In particular if your data store is a relational DB then SQLite is a fantastic option. Just remember that there are differences between SQLite and other popular relational DBs like MySQL and PostgreSQL.

Make sure you account for that in your tests. Note that you still access your data through the abstract data layer, but now the backing store during tests is the in-memory data store. Your test will populate the test data differently, but the code under test is blissfully unaware of what's going on.

Using SQLite

SQLite is an embedded DB (linked with your application). There is no separate DB server running. It typically stores the data in a file, but also has the option of an in-memory backing store. 

Here is the InMemoryDataStore struct. It is also part of the concrete_data_layer package, and it imports the go-sqlite3 third-party package that implements the standard Golang "database/sql" interface.  

Constructing the In-Memory Data Layer

The NewInMemoryDataLayer() constructor function creates an in-memory sqlite DB and returns a pointer to the InMemoryDataLayer

Note that each time you open a new ":memory:" DB, you start from scratch. If you want persistence across multiple calls to NewInMemoryDataLayer(), you should use file::memory:?cache=shared. See this GitHub discussion thread for more details.

The InMemoryDataLayer implements the DataLayer interface and actually stores the data with correct relationships in its sqlite database. In order to do that, we first need to create a proper schema, which is exactly the job of the createSqliteSchema() function in the constructor. It creates three data tables—song, user, and label—and two cross-reference tables, label_song and user_song.

It adds some constraints, indexes, and foreign keys to relate the tables to each other. I will not dwell on the specific details. The gist of it is that the entire schema DDL is declared as a single string (consisting of multiple DDL statements) that are then executed using the db.Exec() method, and if anything goes wrong, it returns an error. 

It's important to realize that while SQL is standard, each database management system (DBMS) has its own flavor, and the exact schema definition will not necessarily work as is for another DB.

Implementing the In-Memory Data Layer

To give you a taste of the implementation effort of an in-memory data layer, here are a couple of methods: AddSong() and GetSongsByUser()

The AddSong() method does a lot of work. It inserts a record into the song table as well as into each of the reference tables: label_song and user_song. At each point, if any operation fails, it just returns an error. I don't use any transactions because it is designed for testing purposes only, and I don't worry about partial data in the DB.

The GetSongsByUser() uses a join + sub-select from the user_song cross-reference to return songs for a specific user. It uses the Query() methods and then later scans each row to populate a Song struct from the domain object model and return a slice of songs. The low-level implementation as a relational DB is hidden safely.

This is a great example of utilizing a real relational DB like sqlite for implementing the in-memory data store vs. rolling our own, which would require keeping maps and ensuring all the book-keeping is correct. 

Running Tests Against SQLite

Now that we have a proper in-memory data layer, let's have a look at the tests. I placed these tests in a separate package called sqlite_test, and I import locally the abstract data layer (the domain model), the concrete data layer (to create the in-memory data layer), and the song manager (the code under test). I also prepare two songs for the tests from the sensational Panamanian artist El Chombo!

Test methods create a new in-memory data layer to start from scratch and can now call methods on the data layer to prepare the test environment. When everything is set up, they can invoke the song manager methods and later verify that the data layer contains the expected state.

For example, the AddSong_Success() test method creates a user, adds a song using the song manager's AddSong() method, and verifies that later calling GetSongsByUser() returns the added song. It then adds another song and verifies again.

The TestAddSong_Duplicate() test method is similar, but instead of adding a new song the second time, it adds the same song, which results in a duplicate song error:


In this tutorial, we implemented an in-memory data layer based on SQLite, populated an in-memory SQLite database with test data, and utilized the in-memory data layer to test the application.

In part three, we will focus on testing against a local complex data layer that consists of multiple data stores (a relational DB and a Redis cache). Stay tuned.

Did you find this post useful?
Want a weekly email summary?
Subscribe below and we’ll send you a weekly email summary of all new Code tutorials. Never miss out on learning about the next big thing.
Looking for something to help kick start your next project?
Envato Market has a range of items for sale to help get you started.