Because read is a user-mode program, and then every time read corresponds to a system call (switching from user mode to kernel mode and then back to user mode), in fact, the most time-consuming thing is state switching.
If you read five bytes at a time, there is only one system call (two state switches).
If you read one byte at a time and need to read it five times, there will be five system calls (10 state switching).
The specific difference is related to the actual environment, and a lot of data is needed to see it when testing.