Dynamic databases connections with NestJS

The official NestJS documentation explains very clearly how to handle multiple database connections in a static configuration.

One of our clients has a different use-case and wants the application to be able to connect to many databases, each one dedicated to a country. He wants to add or remove a database dynamically without changing the codebase.

Here are the main differences between these two models :

 StaticDynamic
Database structureSame or differentHas to be the same
Add / remove databaseCodebase changeConfiguration change and application restart

How to setup dynamic configuration

Here is the .env file of what we want to achieve :

DATABASE_SYSTEM_IDS=FR,DE,GB

DB_FR_TYPE=postgres
DB_FR_HOST=localhost
DB_FR_PORT=5432
DB_FR_USERNAME=fr
DB_FR_PASSWORD=fr
DB_FR_DATABASE=fr
DB_FR_SYNCHRONIZE=true

DB_DE_TYPE=postgres
DB_DE_HOST=localhost
DB_DE_PORT=5433
DB_DE_USERNAME=de
DB_DE_PASSWORD=de
DB_DE_DATABASE=de
DB_DE_SYNCHRONIZE=true

DB_GB_TYPE=postgres
DB_GB_HOST=localhost
DB_GB_PORT=5434
DB_GB_USERNAME=gb
DB_GB_PASSWORD=gb
DB_GB_DATABASE=gb
DB_GB_SYNCHRONIZE=true 

DATABASE_SYSTEM_IDS is a coma separated list of arbitrary defined system identifiers to all the databases we want to connect to. For each one of them, we have DB_xx_* variables where xx is the database system identifier.

Below is the configuration class which can handle this kind of configuration (file src/common/config/orm.config.ts) :

import { registerAs } from '@nestjs/config';
import * as dotenv from 'dotenv';

dotenv.config(); // used to get process.env access prior to AppModule instanciation

export const getDatabaseSystemIds = (): string[] => {
  return process.env.DATABASE_SYSTEM_IDS.split(',');
};

export default registerAs('orm', () => {
  const config = {};

  getDatabaseSystemIds().forEach((systemId) => {
    config[systemId] = {
      type: process.env[`DB_${systemId}_TYPE`],
      host: process.env[`DB_${systemId}_HOST`],
      port: parseInt(process.env[`DB_${systemId}_PORT`]),
      username: process.env[`DB_${systemId}_USERNAME`],
      password: process.env[`DB_${systemId}_PASSWORD`],
      database: process.env[`DB_${systemId}_DATABASE`],
      synchronize: process.env[`DB_${systemId}_SYNCHRONIZE`] === 'true',
      entities: [`${__dirname}/.. /.. /**/*.entity{.ts,.js}`],
    };
  });

  return config;
}); 

Here is how we generate dynamically a TypeORM configuration per database (file src/home/jnesis/www/wp.module.ts) :

import { Module } from '@nestjs/common';
import { ConfigModule, ConfigService } from '@nestjs/config';
import { TypeOrmModule } from '@nestjs/typeorm';
import { CarManufacturersModule } from './car-manufacturers/car-manufacturers.module';
import ormConfig, { getDatabaseSystemIds } from './common/config/orm.config';

// database connection for each system id
const databasesConfig = getDatabaseSystemIds().map((systemId) => {
  return TypeOrmModule.forRootAsync({
    name: `database-${systemId}`,
    imports: [ConfigModule.forFeature(ormConfig)],
    useFactory: (config: ConfigService) => config.get(`orm.${systemId}`),
    inject: [ConfigService],
  });
});

@Module({
  imports: [
    ConfigModule.forRoot({
      isGlobal: true,
    }),
    ...databasesConfig,
    CarManufacturersModule,
  ],
  controllers: [],
  providers: [],
})
export class AppModule {} 

Note each one of these TypeORM configurations is assigned with the name database-xx where xx is the database system identifier.

How to setup dynamic dependency injection

Now our configuration is dynamic, we need to be able to connect to a specific database through its system identifier. Standard static dependency injection through (@InjectEntityManager , @InjectRepository, etc…) cannot be used as it requires to know the database we want to connect to at compile time.

 

To be able to dynamically select a database on runtime, we need to use the module reference feature from NestJS. This way, we can retrieve an EntityManager from its TypeORM configuration name database-xx (see our loadEntityManager method in file src/car-manufacturers/car-manufacturers.service.ts) :

import { Injectable } from '@nestjs/common';
import { ModuleRef } from '@nestjs/core';
import { getEntityManagerToken } from '@nestjs/typeorm';
import { EntityManager } from 'typeorm';
import { CarManufacturer } from './car-manufacturer.entity';

@Injectable()
export class CarManufacturersService {
  constructor(private moduleRef: ModuleRef) {}

  private async loadEntityManager(systemId: string): Promise<EntityManager> {
    return this.moduleRef.get(getEntityManagerToken(`database-${systemId}`), {
      strict: false,
    });
  }

  async findAll(countryCode: string): Promise<CarManufacturer[]> {
    const entityManager = await this.loadEntityManager(countryCode);
    if (!entityManager) {
      return [];
    }

    return entityManager.find(CarManufacturer);
  }
} 

Demo project

You’ll find the complete working example project here in Github.

Same database structure limitation

Actually we could have different structure in dynamic databases. It requires to define the shape of each database specifying the set of entities it uses.

As an example we could add a parameter to the .env file :

DB_FR_ENTITIES=structureA

DB_DE_ENTITIES=structureB

DB_GB_ENTITIES=structureA 

And then dynamically load entities depending using the structure name as a suffix in file src/common/config/orm.config.ts :

entities: [`${__dirname}/.. /.. /**/*.${process.env[`DB_${systemId}_ENTITIES`]}.entity{.ts,.js}`], 

You’ll have to define entities for each database structure, and they can be completely different :

  • StructureA: car-manufacturer.structureA.entity.ts
  • StructureB: brand.structureB.entity.ts, address.structureB.entity.ts

Obviously the database queries will also be different from one database structure to another, so it will require additional conditions and branching in the business code.

2 Responses

  1. Hi,
    excellent content. Could you please provide a full CRUD Operations of this?
    I’m having trouble getting the CRUD Operations working with this method since I couldn’t find any documentation of it.

    1. Hi,

      Thank you for your feedback !

      Once you get that EntityManager instance (as shown in CarManufacturersService) you can do any regular CRUD and other database operations you want.

      Have a look at this :
      – EntityManager introduction: https://typeorm.io/working-with-entity-manager
      – EntityManager API: https://typeorm.io/entity-manager-api#entitymanager-api

      Please note you could get a Repository instance instead of an EntityManager replacing “getEntityManagerToken(…)” method with “getRepositoryToken(…)”.

Leave a Reply

Your email address will not be published. Required fields are marked *