How to create custom cache server for different APIs?
This post was translated using the DeepL translator, so I apologize for any incomprehensibility. In this article you can read more about my decision.
If we want to display posts from social networks on our website or any other website, or work with an API, it's often a good idea to have your own server that caches the results from the original API. This is especially true when queries to the API are limited and there is a real risk that one one day the server for that service will shut down and not send anything.
In this article I would like to discuss how to create such a simple API cache using Express.js application. So I assume at least basic knowledge of JavaScript, working with NPM (Node Package Manager) and command line.
Getting Started
The easiest way to get started with an Express.js application is to use the Express application generator. This is simple tool that prepares the basic structure of the project and adds information about the libraries that we will need for our application. So let's start by creating a folder with our application:
1mkdir API-Cache && cd API-Cache
Then we create a new Express.js application in the folder and install some libraries. I also recommend to take a look for possible configuration options in the documentation.
1npx express-generator && npm install
This should create the basic structure of the project in our folder, including the node_modules
folder containing all the libraries
specified in the package.json
file. For our project, however, we will need a few more libraries that
generator does not contain, namely dotenv, cors and got, and possibly also nodemon to automatically restart the server when there is a change in the code.
1npm install --save dotenv cors got 2# since we don't need nodemon in production, we only install it as a dev dependency 3npm install --save-dev nodemon
Finally, we create an empty .env
file that will contain sensitive data of our application, such as authentication tokens.
1touch .env
This should give us everything we need to develop the application.
Information source
For our example application, we will use the Quotes on Desing API, which is freely available, as a source of information. contains a number of quotes from famous people. The API information can be found here for our purposes, copy this link:
https://quotesondesign.com/wp-json/wp/v2/posts/?orderby=rand&per_page=1
The results are sorted randomly, and due to the per_page=1
parameter, we only ever see one quote.
Cache
Next we need to solve the cache of the provider, in this article I will work with the cache in the server memory. It is not ideal solution, especially if you need to store a larger amount of data or if the data should remain in the cache even after a server reboot. In this case, it would be better to choose a cache that is stored on disk. However, in this case I assume, the data will only stay in the cache for a few tens of minutes at most.
So we will create a simple constructor function/class that will hold our data. So we'll create a folder called mkdir utils
in which we'll
to store our constructor function/class - DataCache.js
.
DataCache.js
1function DataCache (fetchFunction, minutesToLive = 10) { 2 this.fetchFunction = fetchFunction; 3 this.millisecondsToLive = minutesToLive * 60 * 1000; 4 this.fetchDate = new Date(0); 5 this.cache = null; 6 7 //... 8 9} 10 11module.exports = DataCache;
The basic form of our constructor function/class DataCache is as follows, it accepts two parameters.
The first fetchFunction
is a method to fetch data from our source when the cache needs to be updated.
The second parameter is the lifetime of the data in cache before it is no longer valid. For clarity, we will specify it in minutes
and we'll convert them to milliseconds afterwards. Next, we prepare a variable this.cache
which should contain the data and this.fetchDate
,
which will carry the date the data was last read into cache.
We will also add the isCacheExpired()
function to our class to find out whether cache has expired or not.
1 this.isCacheExpired = () => { 2 return (this.fetchDate.getTime() + this.millisecondsToLive) < new Date().getTime(); 3 }
The method compares the current time with the sum of the time stored in the variable
fetchDate
and the cache lifetime. If this time is less than the current time, cache expires.
The most important function, however, is getData()
, as it provides both the retrieval of data from the source and the return of data from the cache, namely
depending on whether it is cached and still valid.
1 this.getData = () => { 2 if (!this.cache || this.isCacheExpired()) { 3 console.log('expirováno - načítám nová data'); 4 return this.fetchFunction() 5 .then((data) => { 6 this.cache = data; 7 this.fetchDate = new Date(); 8 return data; 9 }); 10 } else { 11 console.log('data jsou v cache'); 12 return Promise.resolve(this.cache); 13 } 14 }
The method always returns Promise with data. For better clarity, I also left the logs in the code, which we can see in the command line to show if the server has read the data from cache or from cache. This completes our DataCache class and we can try it out in practice.
Loading data on the server
So we create a controllers folder in the project root mkdir controllers && cd controllers
and in it a controller touch QuoteController.js
. The QuoteController will contain only one method, namely getQuote
, which will query the Quote on Design server for the current quote.
QuoteController.js
1const got = require('got'); 2const DataCache = require('../utils/DataCache'); 3 4function getQuote () { 5 return got(`https://quotesondesign.com/wp-json/wp/v2/posts/?orderby=rand&per_page=1`) 6 .then((response) => { 7 return JSON.parse(response.body); 8 }) 9 .catch((error) => { 10 return error; 11 }); 12} 13 14const quote = new DataCache(getQuote, 1); 15 16module.exports = quote;
First we load the necessary libraries, which is the got we will need for
API query and our DataCache class. Then
we declare a function getQuote()
which consists of just a query to the API. For the query
we use the got library, even though it would be
could also use the standard HTTP module, but got gives us a much
saves us a lot of work. In the then
block, we also convert the response body to JSON format,
since got itself does not handle the response and only returns a text string.
Finally, we create a variable in which we initialize our DataCache class and export it.
We add the following lines to the ./routes/index.js
file. We will add Require to our controller below the others
require lines. Then register our path anywhere below the original code, but before module.exports
.
1// ... original require statements 2const quoteController = require('../controllers/quoteController'); 3 4// ... original code 5router.get('/quote', async (req, res, next) => { 6 quoteController.getData().then((data) => { 7 res.json(data); 8 }); 9});
In the route registration we just call controller and in it a method from our class that mediates for us
data either from cache or from a query to the API. We then send the returned data in json format. That's it! Now
just run the npm run start
command and visit the http://localhost:3000/quote
page in your browser, where you should
you should see a random quote! You should see the following in the server console:
[nodemon] starting `node ./bin/www`
expired - loading new data
GET /quote 200 164.028 ms - 5056
data is cached
GET /quote 200 1.710 ms - 5056
data is in cache
GET /quote 200 0.969 ms - 5056
Conclusion
Congratulations on successfully creating a very simple and basic cache API! If you have decided that you
such a simple cache is enough. I recommend you take a look at the whole code again and extend or modify the error trapping as well
it is probably advisable to remove the console.log()
parts of the code. The DataCache
class can also be extended very easily with the fs
module.
module with the ability to cache to disk, although even this may not be ideal in some cases.