Simply changing to !ID did not solve the type issue:

UnhandledPromiseRejectionWarning: GraphqlError: Type mismatch on variable $package_id and argument packageVersionId (String! / ID!)

See failed run

We are POST'ing to the endpoint itself, which might get rate-limited

Tested on in runtime-components using 1.0.3-dev1 See run

Simply changing to !ID did not solve the type issue:

UnhandledPromiseRejectionWarning: GraphqlError: Type mismatch on variable $package_id and argument packageVersionId (String! / ID!)

See failed run

We are POST'ing to the endpoint itself, which might get rate-limited

Tested on in runtime-components using 1.0.3-dev1 See run

Simply changing to !ID did not solve the type issue:

UnhandledPromiseRejectionWarning: GraphqlError: Type mismatch on variable $package_id and argument packageVersionId (String! / ID!)

See failed run

We are POST'ing to the endpoint itself, which might get rate-limited

Tested on in runtime-components using 1.0.3-dev1 See run

Notable changes:

• Removed current concurrency + barrier logic for batch processing
• Created RunnerPool, which spawns partitionsConsumedConcurrently number of independent runners
• Created FetchManager, which fetches different brokers independently and assigns partitions to different runners during the consumer group sync
• Each connection now opens two sockets - one for fetch requests, the other one for the rest (TCP sockets preserve the order of messages sent to a broker. That means that other concurrent requests, eg. heartbeats and offset fetch/commit, from other runners would wait behind the fetch polls and drastically slows down the consumption in the concurrent scenario)
• Removed memory leaks from hanging consumers in tests

Manually compared consumption times with the scripts below:

Producer:

const { Kafka } = require('kafkajs');

const kafka = new Kafka({ brokers: ['localhost:9092'] });

const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

(async () => {
    // const admin = kafka.admin();
    // await admin.connect()
    // await admin.deleteTopics({ topics: ['loadtest'] })
    // await admin.createTopics({ topics: [{ topic: 'loadtest', numPartitions: 3, replicationFactor: 1 }]})
    // await admin.disconnect()

    const producer = kafka.producer();
    await producer.connect();

    const batchSize = 100;
    for (let i = 0; i < 5; i++) {
        const messages = Array(batchSize)
            .fill()
            .map((_, index) => ({
                key: `key-${batchSize * i + index}`,
                value: `value-${batchSize * i + index}`,
            }));
        await producer.send({ topic: 'loadtest', messages });
        await sleep(1000);
    }
    await producer.disconnect();

    console.log('Published!');
})();

Consumer:

const { Kafka } = require('kafkajs');

const kafka = new Kafka({ brokers: ['localhost:9092'] });

const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

(async () => {
    const metrics = {};

    const consumer = kafka.consumer({ groupId: 'tulios-consumer', maxWaitTimeInMs: 5000 });
    await consumer.connect();

    process.on('SIGINT', () => {
        Object.entries(metrics).forEach(([partition, { startTime, endTime }]) => {
            console.log(`Partition ${partition} processed in ${endTime - startTime} ms`)
        })
        return consumer.disconnect()
    });

    await consumer.subscribe({ topic: 'loadtest' });
    await consumer.run({
        partitionsConsumedConcurrently: 3,
        eachMessage: async ({ partition, message: { value } }) => {
            if (!metrics[partition]) metrics[partition] = { startTime: Date.now() }

            if (partition === 0) await sleep(100)
            console.log(partition, value.toString());

            metrics[partition].endTime = Date.now()
        },
    });
})();

Slower runners now consume their batches at their own pace, other fetch requests will not be blocked. Partitions 2 and 3 were consumed in ~70ms (- 4s from delays) compared to the current master version, which took ~9s (-4s delays):