feat: prohibit guests from surpassing fd limits by n0toose · Pull Request #1247 · hermit-os/uhyve

n0toose · 2026-02-08T17:17:31Z

This allows Uhyve to gracefully handle file descriptor exhaustion
which can be caused by a busy host or a malicious guest.
Introduction of a serial test (which also implicitly tests whether
the errno provided by the host is 'parsed' properly). Not
complete, but still something.
Dynamic calculation of available files for the guest (as in,
files used by Uhyve are not included in the 'files left' variable).

Fixes #897
Depends on #1219

Currently Linux-only.

Co-authored-by: Panagiotis "Ivory" Vasilopoulos <git@n0toose.net>

- bump uhyve-interface to 0.2.0 - perform virtaddr -> physaddr conversions in hermit for read, write. - use adjust serial_port based on version of uhyve-interface used - Exit hypercalls should encode the exit code instead of a pointer to the integer. ExitArgs has been removed. - v2: add v2 support in src/stats.rs, futureproof - v2: use u64, i64 instead of usize, isize - v2: improve errno returns in read, write hypercalls with v1 compat - v2: remove packed where unnecessary - v2: return better error (codes) for write, read - v2: return length of write for stdout, stderr prints - adjust serial test for v2 uhyve-interface (v2) should now write to the rcx register before triggering an IoOut hypercall on systems the x86_64 architecture so as to allow for using pointers to 64-bit addresses (ports restrict us to 32-bit). This restriction should not apply for serial writes, although it also remains possible for the kernel to print out messages. Co-authored-by: Ellen Emilia Anna Zscheile <fogti+devel@ytrizja.de> Co-authored-by: Jonathan Klimt <jonathan.klimt@eonerc.rwth-aachen.de>

... which causes excessive slowdowns at scale.

In some cases, the host may return errors that 'Hermit' doesn't "know about". This warning intends to aid with the debugging process.

- write_to_fd test is now part of fs_tests - standard streams' (written to by write_to_fd) output is checked - generate_params now takes Option<PathBuf> as guest_path, as such a path may not always be necessary (like in the case of write_to_fd). A superfluous guest_path was also removed from mounts_test.

- This allows Uhyve to gracefully handle file descriptor exhaustion which can be caused by a busy host or a malicious guest. - Introduction of a serial test (which also implicitly tests whether the errno provided by the host is 'parsed' properly). Not complete, but still something. - Dynamic calculation of available files for the guest (as in, files used by Uhyve are not included in the 'files left' variable). Fixes hermit-os#897

n0toose · 2026-02-08T17:49:55Z

No idea why the integration tests that run locally (after having close to an hour trying to "engineer" that) just don't feel like running on GitHub Actions right now.

fogti · 2026-02-08T21:51:57Z

Catching and handling the errors (taken from errno -l output):

ENFILE 23 Too many open files in system
EMFILE 24 Too many open files

Might be the cleaner solution, in particular because they aren't Linux-specific.
For Linux, we could still provide something linux-specific additional debug output, but the core functionality would optimally only depend on POSIX / UNIX,

fogti · 2026-02-08T21:59:11Z

Also, the amount of places where fd limits can be imposed is frightingly large: https://www.baeldung.com/linux/error-too-many-open-files . And Linux occasionally adds new ones.

n0toose · 2026-02-08T23:07:38Z

I didn't find polling lsof or similar to be a good idea and explicitly did not touch anything involving setting those limitations

Might be the cleaner solution, in particular because they aren't Linux-specific.

Those were the codes I'd theoretically return, not part of the Hermit ABI.

handling the errors

One could do that (and I should've mentioned it earlier), but this limits the possibility of being able to define arbitrary limits (e.g. max open files, enforced by the kernel)—the idea would be useful for an "same process, many guests" approach later. This is not part of this PR, but that could change. It is in line with plans to decide whether errors cause crashes, or whether a host mount is read only.

It might make sense to do both, but not at the same time, to ensure that there is some parity e.g. on macOS. After all, it's better if Uhyve doesn't treat the underlying guests differently depending on the host OS—but it is completely fair for Uhyve to provide some additional features depending on the host OS (i.e. Landlock) or configuration (e.g. tracing).

n0toose · 2026-02-08T23:12:15Z

It seems like the many methods effectively affect the same thing, like how sysctl may affect the same thing as a parameter passed to the kernel using a bootloader by GRUB.

Unless if I'm missing something (e.g. ulimit?), the primary thing that I don't do is use /proc/sys/fs/file-max, but I considered that reading it is wasteful compared to just setting a MAX value. One would get an error before you reach either of these values, unless if you decide to get Uhyve to run as the init system or something like that (other processes will use file descriptors and we don't know how many).

jounathaen · 2026-02-09T09:28:51Z

Cargo.toml

 criterion = "0.8"
 home = "0.5"
 regex = { version = "1.12.2", default-features = false, features = ["unicode-perl"] }
+serial_test = "3.3"


This is probably something for a separate PR

I'm not decided if I prefer another dependency, or if we should just use a global Mutex here, similar to what is done in build_hermit_bin

jounathaen · 2026-02-09T09:30:23Z

tests/test-kernels/Cargo.toml


 [target.'cfg(target_os = "hermit")'.dependencies]
-hermit = "0.12"
+hermit = { path = "../../../hermit-rs/hermit" }


fogti · 2026-02-09T09:50:19Z

I think on Linux it might make more sense to set ulimit explicitly inside of uhyve, and rely on the outer kernel for enforcement.

The errno codes mentioned are those that would be emitted by syscalls (e.g. open) to the outer kernel when trying to surpass limits.

fogti · 2026-02-09T09:57:35Z

btw. Mac OSX also supports ulimit afaik: https://wilsonmar.github.io/maximum-limits/

I don't think it makes sense to reinvent the wheel for stuff that can be done with standard UNIX utilities just fine.

fogti · 2026-02-09T10:57:48Z

Consider using https://docs.rs/rlimit/ instead of hacking around with non-portable /proc entries.
That can also be used to set resource limits.

fogti · 2026-02-09T11:05:08Z

After a guest has started running, do we even open any additional file handles which would necessitate such a paranoid implementation?

The main thing that we might want to fix is the handling of rftrace trace writing stuff, which afaik does open file descriptors in a manner that might fail under strict resource limits. Is there anything else that isn't guest-directed?

n0toose · 2026-02-09T14:41:08Z

The main thing that we might want to fix is the handling of rftrace trace writing stuff, which afaik does open file descriptors in a manner that might fail under strict resource limits.

This has not been tested with rftrace, but the current implementation checks the current amount of open file descriptors while UhyveVm is being instantiated. The only negative aspect of this right now is that a refactor (e.g. closing the ELF header before UhyveFileMap is initialized) could break the logic.

ulimit

TBD, thanks. Will leave this on the backburner until v2 is merged.

jounathaen and others added 6 commits February 8, 2026 11:56

feat(interface): initial version -> v1 folder

9217e64

Co-authored-by: Panagiotis "Ivory" Vasilopoulos <git@n0toose.net>

docs(interface): add README file

110044d

feat(v2): use sync_regs to avoid ioctl on x86_64

15c44d9

... which causes excessive slowdowns at scale.

feat(hypercall): show warning if errno can't be translated

7b9dfad

In some cases, the host may return errors that 'Hermit' doesn't "know about". This warning intends to aid with the debugging process.

n0toose requested review from fogti and jounathaen February 8, 2026 17:17

n0toose added feature/file system feature/security Concerns security-related behavior, soundness, isolation or reliability. tests Integration and unit-tests verifying Uhyve's behavior labels Feb 8, 2026

n0toose force-pushed the fd-limits branch from 78f4d61 to 182bcf8 Compare February 8, 2026 17:19

jounathaen reviewed Feb 9, 2026

View reviewed changes

n0toose marked this pull request as draft February 17, 2026 16:01

Conversation

n0toose commented Feb 8, 2026

Uh oh!

n0toose commented Feb 8, 2026

Uh oh!

fogti commented Feb 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fogti commented Feb 8, 2026

Uh oh!

n0toose commented Feb 8, 2026

Uh oh!

n0toose commented Feb 8, 2026

Uh oh!

jounathaen Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

jounathaen Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

n0toose Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

fogti commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fogti commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fogti commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fogti commented Feb 9, 2026

Uh oh!

n0toose commented Feb 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fogti commented Feb 8, 2026 •

edited

Loading

fogti commented Feb 9, 2026 •

edited

Loading

fogti commented Feb 9, 2026 •

edited

Loading

fogti commented Feb 9, 2026 •

edited

Loading