I have noted errors “OS Error 61: No data” on some file-heavy workloads on AWS FSX Lustre. They have also been encountered in this bug report.

Symptom

The same error can be seen in Python programs reporting:

OSError: [Errno 61] No data available:

Or in rust programs reporting:

No data available (os error 61)

In both systems the error is encountered only when copying a file from the Lustre filesystem.

Underlying cause

The immediate cause is use of the sendfile syscall. Very occasionally when reading from the Lustre filesystem the call returns ENODATA and neither the Python nor the rust libraries fall back on standard read()/write() but instead throw an exception.

copy_file_range(3, NULL, 4, NULL, 1073741824, 0) = -1 EXDEV (Invalid cross-device link)
sendfile(4, 3, NULL, 2147479552)        = -1 ENODATA (No data available)
close(4)                                = 0
close(3)                                = 0
write(2, "Error: ", 7Error: )                  = 7
write(2, "Os", 2Os)                       = 2

Workaround

On Python the use of sendfile for copy can be disabled at runtime, see shutils.

Motivation

Use of sendfile() instead of read()/write() is motived by the performance improvement arising from avoiding the need to copy data to userspace and back to the kernel.

results

Other relevant infomration

  • sendfile issue on another distributed filesystem: https://bugs.python.org/issue43743#msg393429