One step forward, one step... sideways?... with my steem rewards Lorenz Curves project

Since my last update on my Node.js Lorenz Curves project I made some progress. I implemented a rate-limiter module in my code so I wasn't bombarding any servers with api requests. It took a bit of effort in order to figure out how to think of this in software terms -- because of my background in computer hardware and low-level programming I often find it difficult to remap how I "naturally" think of problems into how things are done in a high-level language. Conceptually it seemed really simple to have things go into a queue from which they would get dispatched, but when the "things going into the queue" are function calls which can pause in the middle until a promise is resolved it took a bit for that concept to gel into actual code for me. But I think this works:

// Rate limiting function.
let lastBurstStart = new Date();
let numRequestsInBurst = 0;
let maxRequestsPerBurst = 5;
let timePerBurst = 5000; // in milliseconds
let lowPriorityQueue = [];
let highPriorityQueue = [];
function clearRateLimitWindow() {
  numRequestsInBurst = 0;
  lastBurstStart = new Date();
  let numHighRequestsToIssue = Math.min(highPriorityQueue.length, maxRequestsPerBurst);
  for (let i=0; i<numHighRequestsToIssue; i++) {
    let unqueuedResolve = highPriorityQueue.shift();
    unqueuedResolve(true);
    numRequestsInBurst++;
  }
  let numLowRequestsToIssue = Math.min(lowPriorityQueue.length, maxRequestsPerBurst-numHighRequestsToIssue);
  for (let i=0; i<numLowRequestsToIssue; i++) {
    let unqueuedResolve = lowPriorityQueue.shift();
    unqueuedResolve(true);
    numRequestsInBurst++;
  }

  //console.log("RateLimiter: Dispatched", numRequestsInBurst, "at", new Date());

  // if we issued requests we need to schedule this function to clear the window
  if (numRequestsInBurst > 0) {
    setTimeout(clearRateLimitWindow, timePerBurst);
  }
}
function getTicket(lowPriority=false) {
  let ticketPromise; // = new Promise();
  if (numRequestsInBurst == 0) {
    // start of a new burst.
    lastBurstStart = new Date();
    numRequestsInBurst = 1;
    ticketPromise = Promise.resolve(true); // don't wait, caller is good to go

    // schedule a process to clear # of requests in current window.
    setTimeout(clearRateLimitWindow, timePerBurst);
  } else if (numRequestsInBurst < maxRequestsPerBurst) {
    // we're in the midst of a burst but not yet at the limit.
    numRequestsInBurst++;
    ticketPromise = Promise.resolve(true); // don't wait, caller is good to go
  } else {
    // we've sent our limit, this one needs to be queued.
    ticketPromise = new Promise((resolve, reject) => {
      if (lowPriority) {
        lowPriorityQueue.push(resolve);
      } else {
        highPriorityQueue.push(resolve);
      }
    });
  }

  return ticketPromise;
}

module.exports = {
  getTicket,
  maxRequestsPerBurst,
  timePerBurst
};

So I pulled that into my earlier code that queries the Steem database for the transactions in a block so that it can pull out the virtual transactions (which are the ones that give rewards), and eventually they'd get aggregated into the curves for my project. But when I started running some tests to see what I should set my rate-limit parameters to in order to avoid getting database errors, it seemed like I was only getting an average of about one response per second from the api calls. So if that's the real limit it's a bit of a problem, since the chain adds a new block every 3 seconds and I was hoping to get data for months or years worth of blocks, which would translate to waiting days or months for all of the data to get back to me before I could do any processing. That doesn't seem practical.

I'm guessing that there would be a way for me to get the data without the bottleneck if I investigated what "running a node" involves (from skimming a post or two I gather that's how steemDB did it), but I'm skeptical that I have the resources in terms of things like disk space right now, even if I was prepared to try to figure out the technical details. (And disk space was likely going to turn into a problem with my "local cache" implementation anyway). So I'm not sure if or how I can proceed with this project. One thing I am considering is that this was supposed to be my "practice project" before considering some gaming related ideas, so I may just put this one on the shelf for now and try to work on one of those. But I haven't fully decided yet.

(image from Pixabay)