I used the log module when I wrote the Node.js project this time, and I encountered a small problem.
This is a project that performs configurable automation tasks regularly, so the output information will increase continuously, which means that the log files will increase continuously over time. If the log file size is not controlled, the server's disk will be full sooner or later. So it is necessary to limit the file size.
The ideal control method is to clear the first recorded data when the file size exceeds the limit. Similar to a FIFO queue.
# Delete previous data
- 1 xxx
......
100 abc
# Append data at the end of the file
+ 10 1 xxxx
File scrolling of log4js
Speaking of logs, many Node.js developers will definitely look for log4js. Let's take a look at how log4js handles this problem.
Log4js is divided into many appenders (which can be understood as media for recording logs), and the file scrolling function can be configured through functions.
There are two ways to scroll files: date and file size.
To control the file size, of course, choose the latter.
In order to test whether this function meets our requirements, write a loop code and write a log.
const log4js = require('log4js ')
//configure log4js
log4js.configure({
Appendix: {
Everything: {
Type: File,
File name:' a.log',
maxLogSize: 1000，
Backup: 0
},
},
Category: {
Default value: {
Appenders: ['everything'],
Level: "Debugging"
}
}
});
const log = log 4js . get logger()；
For (let I = 0；; I<4 1; i++) {
const str = i.toString()。 padStart(6，' 000000 ')；
log . debug(str)；
}
After execution, two files a.log and a.log. 1 will be generated.
In which a.log. 1 has 20 rows of data, and the actual size is 1kb, while a.log only has 1 row of data.
Although it does control the file size, it brings two problems:
An extra backup file will be generated, and the total disk space occupied will exceed the file limit.
The size of the log file is variable, so it is likely to be necessary to query the log in combination with the backup file (for example, the above situation log file only has 1 line data).
It is speculated that the implementation logic of log4js may be as follows:
Check whether the log file reaches the limit size, if so, delete the backup file, otherwise, continue to write the log file.
Rename the log file to a backup file.
This obviously cannot fully meet the demand.
String replacement?
If you want to complete the loop overwrite operation in memory, it is relatively simple, just use string or Buffer.
Add the length of the string/buffer, and intercept it if it exceeds the size.
Write and overwrite the log file.
But there is a big problem: it takes up memory.
For example, if the file size is limited to 1GB and 10 log files are written at the same time, at least 10GB of memory space will be occupied!
Memory is more valuable than disk space, and such an obvious performance problem is obviously not the best solution.
File volume
According to the requirements, the implementation steps can be divided into two steps:
Append the latest data to the end of the file. (the fs module of Node.js has corresponding functions)
Delete the part at the beginning of the file that exceeds the limit. (Node.js has no response function)
These two steps are in no particular order, but Node.js does not provide an API to delete the beginning of the file, but only provides a function to modify the specified location of the file.
Since you can't delete the contents at the beginning of the file, change your mind and only keep the contents at the end of the file (within the size limit).
What? Isn't this the same thing?
Slightly different ~
Delete is the operation of the original file, while the reserved content can be operated with the help of temporary files.
So the idea became:
Create a temporary file whose contents come from the log file.
Add data to a temporary file.
The contents that meet the file size limit in the temporary file are read from back to front (in the form of offset) and copied to the log file for overwriting.
In order not to occupy additional disk space, the temporary file will be deleted after the write operation is completed.
In this way, there will be no incomplete log files like log4js, and no additional temporary files will be kept. But the operation of io will increase ~
The write operation can be realized by the tail command, and the final code is as follows:
Private writing (name: string, buf? :Buffer | string) {
//append buf to tmp file
const tmpName = name.replace(/(。 *\/)(.*$)/, '$ 1_\.$2\.tmp’)；
If (! existsSync(tmpName)) {
CopyFileSync (name, tmpname);
}
buf & amp& ampappendFileSync(tmpName，buf)；
//If you are busy, please wait.
If (this.stream & this.stream.readable) {
this . needupdatelog file[name]= true；
} Otherwise {
Try {
execSync(` tail-c $ { limit } $ { tmpName } & gt； $ { name } `)；
Try {
if(this . needupdatelog file[name]){
this . needupdatelog file[name]= false；
This.write (name);
} Otherwise {
ExistsSync(tmpName) and amp & unlinksync (tmpname);
}
} catch (e) {
Console. Error (e);
}
} catch (e) {
Console. Error (e);
}
}
}
abstract
There are two views to accomplish this function:
Quantitative change causes qualitative change. When the amount of data becomes larger, many simple processing methods are not used, such as writing files. If you use writeFile directly, it will take up a lot of memory or even insufficient memory. Therefore, it needs to be split in an appropriate way, and various problems will be encountered in the process of splitting, such as the requirement of intercepting file content in this paper.
Learn to borrow money. Gentleman's nature is the same, and he is also good at faking ~ When a single point cannot complete the operation, it can be achieved through external conditions, such as using temporary files to save the data content in this article.
Well,