The Danger of Synchronous Disbursements

In financial technology and enterprise resource planning (ERP) platforms, executing bulk payroll payouts synchronously during a standard HTTP request-response cycle is an anti-pattern. This approach introduces critical operational failure points:

• Gateway and Connection Timeouts: External payment APIs like Monnify can take several seconds to process a transaction. If processing a bulk array of 100 employees sequentially, the client connection will time out, leaving the transaction state undetermined.
• API Rate Limiting: Sending a burst of synchronous HTTP requests directly to payment gateways often triggers protective rate limits (HTTP 429), resulting in failed partial disbursements.
• The Dual-Write Problem: If the payment gateway succeeds but your local database server crashes before saving the transaction status, your ledger will permanently drift from reality.
• Unmanaged Retries: Simply looping through array elements in an Express handler offers no native state persistence, meaning transient network errors can cause either missed payments or double-payouts.

To solve these architectural liabilities, you must decouple the payment ingestion layer from the execution layer using an asynchronous message queue.

What Is the Decoupled Payroll Architecture?

The architecture splits the task into three independent stages:

1. Ingestion (Producer): An Express API endpoint receives the bulk payroll payout payload, runs structural validation, generates a unique transaction batch, inserts the records into a database with a pending status, and dispatches the task metadata to a persistent queue.
2. Buffer (Queue Store): Redis acts as a high-performance, in-memory data structure store, preserving the tasks and maintaining the strict delivery order of execution states.
3. Execution (Consumer/Worker): A background worker process (utilizing BullMQ) pulls the jobs from the queue at a managed rate, authenticates with the Monnify disbursement gateway, executes the payment using strict idempotency headers, and updates the database status.

Core Concepts and Implementation

1. Decoupling with Express and BullMQ (The Producer)

To establish the queue, configure a connection to your Redis instance and register the BullMQ Queue worker instance.

The following Node.js script demonstrates how to define the Express server, initialize the Redis connection, and structure the route to queue bulk payroll jobs:

const express = require('express');
const { Queue } = require('bullmq');
const IORedis = require('ioredis');

const app = express();
app.use(express.json());

// Establish persistent connection to Redis
const redisConnection = new IORedis({
  host: process.env.REDIS_HOST || '127.0.0.1',
  port: process.env.REDIS_PORT || 6379,
  maxRetriesPerRequest: null
});

// Initialize the payroll queue
const payrollQueue = new Queue('payroll-processing', {
  connection: redisConnection,
  defaultJobOptions: {
    attempts: 3, // Retry failed transactions up to 3 times
    backoff: {
      type: 'exponential',
      delay: 5000 // Start retrying after 5 seconds
    },
    removeOnComplete: true, // Clean up successful logs
    removeOnFail: false // Keep failures for audit
  }
});

// API endpoint to receive bulk payroll payloads
app.post('/api/v1/payroll/disburse', async (req, res) => {
  const { batchId, payments } = req.body;

  if (!batchId || !Array.isArray(payments) || payments.length === 0) {
    return res.status(400).json({ error: 'Invalid batchId or payment array' });
  }

  try {
    // Ingest each payment as an isolated job to prevent complete batch failure
    const jobPromises = payments.map((payment) => {
      const jobId = `${batchId}-${payment.employeeId}`;
      return payrollQueue.add(
        'process-single-payment',
        { batchId, ...payment },
        { jobId } // Use deterministic jobId to enforce idempotency
      );
    });

    await Promise.all(jobPromises);

    return res.status(202).json({
      status: 'ACCEPTED',
      message: 'Payroll batch successfully queued for asynchronous background processing',
      batchId,
      totalRecords: payments.length
    });
  } catch (error) {
    console.error('Failed to queue payroll jobs:', error);
    return res.status(500).json({ error: 'Internal system queuing exception' });
  }
});

app.listen(3000, () => {
  console.log('Ingestion server listening on port 3000');
});

2. Executing Payments Safely (The Consumer Worker)

The consumer worker runs as an isolated process. It pulls execution tasks, negotiates authentication tokens with Monnify, and initiates the payment.

To ensure that a network timeout does not trigger duplicate payments, the worker passes the unique jobId to the Monnify API as an idempotencyKey reference.

const { Worker } = require('bullmq');
const axios = require('axios');

const MONNIFY_BASE_URL = '[https://api.monnify.com/api/v1](https://api.monnify.com/api/v1)';

// Cache for Monnify Auth Token to avoid logging in on every job
let cachedToken = null;
let tokenExpiry = 0;

async function getMonnifyToken() {
  const now = Math.floor(Date.now() / 1000);
  if (cachedToken && now < tokenExpiry) {
    return cachedToken;
  }

  console.log('Authenticating with Monnify API...');
  const authHeader = Buffer.from(
    `${process.env.MONNIFY_API_KEY}:${process.env.MONNIFY_SECRET_KEY}`
  ).toString('base64');

  const response = await axios.post(`${MONNIFY_BASE_URL}/auth/login`, {}, {
    headers: { Authorization: `Basic ${authHeader}` }
  });

  cachedToken = response.data.responseBody.accessToken;
  const expiresIn = response.data.responseBody.expiresIn || 3500;
  tokenExpiry = now + (expiresIn - 100); // Buffer of 100 seconds
  return cachedToken;
}

// Initialize worker to consume queued jobs
const payrollWorker = new Worker('payroll-processing', async (job) => {
  const { batchId, employeeId, amount, bankCode, accountNumber, narration } = job.data;
  
  console.log(`Processing Job ID: ${job.id} for Employee: ${employeeId}`);

  try {
    const accessToken = await getMonnifyToken();

    // Trigger Monnify Single Disbursement API
    const response = await axios.post(
      `${MONNIFY_BASE_URL}/disbursements/single`,
      {
        amount,
        reference: job.id, // Using the deterministic Job ID as the unique external transaction reference
        narration: narration || `Salary Disbursement Batch ${batchId}`,
        destinationBankCode: bankCode,
        destinationAccountNumber: accountNumber,
        sourceAccountNumber: process.env.MONNIFY_SOURCE_ACCOUNT,
        currency: 'NGN'
      },
      {
        headers: {
          Authorization: `Bearer ${accessToken}`,
          'Content-Type': 'application/json'
        }
      }
    );

    const result = response.data;
    if (result.requestSuccessful && result.responseBody.status === 'SUCCESSFUL') {
      console.log(`Success: Payment complete for job ${job.id}`);
      // Perform database state transition here to mark payment as COMPLETED
      return { status: 'COMPLETED', reference: result.responseBody.transactionReference };
    } else {
      throw new Error(`Monnify transaction rejected: ${result.responseMessage}`);
    }
  } catch (error) {
    console.error(`Error processing payment execution for job ${job.id}:`, error.message);
    // Throw exception to trigger BullMQ's automatic backoff and retry pipeline
    throw error;
  }
}, {
  connection: {
    host: process.env.REDIS_HOST || '127.0.0.1',
    port: process.env.REDIS_PORT || 6379
  },
  concurrency: 5 // Process 5 transactions in parallel to respect rate limits safely
});

payrollWorker.on('failed', (job, err) => {
  console.error(`Job ${job.id} definitively failed after all configured attempts:`, err.message);
  // Perform database state transition here to mark payment as FAILED
});

SRE and Architectural Best Practices

• Implement Strict Idempotency: Never reuse references. Construct deterministic job IDs from combinations of unique payroll batches and employee IDs to prevent multiple executions if a task gets double-queued.
• Enforce Concurrency Limits: Monnify limits the volume of simultaneous bulk disbursement calls. Configure your worker concurrency value (concurrency: 5) to prevent rate limits without degrading delivery.
• Implement Dead-Letter Queues (DLQ): Keep failed transactions persisted in Redis rather than auto-removing them. Build an administrative dashboard allowing operators to inspect, correct payload errors, and manually retry failed entries.
• Enable Circuit Breakers: If the Monnify API consistently throws 503 Service Unavailable or network timeouts occur continuously on consecutive jobs, temporarily pause the worker's queue reading loop to allow the provider to recover.

Getting Started

To install and verify this architecture locally:

# Step 1: Spin up local Redis instance
docker run --name local-redis -p 6379:6379 -d redis:alpine

# Step 2: Initialize project and install dependencies
npm init -y
npm install express bullmq ioredis axios dotenv

# Step 3: Run the producer API and the background worker processes
node server.js &
node worker.js

Within minutes, your system will possess a reliable, decoupled financial processing pipeline capable of executing automated, highly-scalable bulk payment disbursements safely.