5kloc is about 10x larger than musl's existing (old) malloc in source lines. I suspect lots of that is low code density, comments, etc.
I have to lookup what exactly mimalloc is/does every time someone mentions it, because the readme/documentation isn't very descriptive except discussing extensions outside the normal API. I didn't have time to dig through this again today. But I did look at it in some depth on several occasions in the past and it really wasn't suitable for or comparable to what we're doing in musl.
We'd be happy to address specific problems on the mailing list. I believe it's a known issue that the Rust compiler is making really heavy use of rapid allocation/freeing cycles, and would benefit from linking a performance-oriented malloc replacement. Doing so is inherently a tradeoff between many factors including performance, memory overhead, safety against erroneous usage by programs, etc.
One statement in your post, which some readers pointed out was apparently added later, "Others have suggested that the performance problems in musl go deeper than that and that there are fundamental issues with threading in musl, potentially making it unsuitable for my use case," seems wrong unless they just meant that the malloc implementation is not thread-caching/thread-local-arena-based. The threads implementation in musl is the only one I'm aware of that doesn't still have significant bugs in some of the synchronization primitives or in cancellation. It's missing a few optional and somewhat obscure features like priority-ceiling mutexes, and Linux doesn't even admit a fully correct implementation in some regards like interaction of thread priorities with some synchronization primitives, but all the basic functionality is there and was written with extreme attention to correctness, and musl aims to be a very good choice in situations where this matters.
The justifications are partly the same as what Daniel Micay has written extensively on in the rational for hardened_malloc (https://github.com/GrapheneOS/hardened_malloc) - unsynchronized per-thread state inherently sacrifices global consistency for performance and makes it impossible to detect a lot of types of memory usage errors (DF/UAF, etc) that could otherwise be caught.
However musl has the additional constraint of being compatible with small/very-low-memory environments. Lack of global consistency inherently means you will end up using memory less efficiently and requesting significantly more from the system. The new malloc about to go upstream in musl is, to my knowledge, the first/only advanced hardened allocator using slab-type design rather than traditional dlmalloc type split/merge, but also designed for extremely low overhead/waste at low to moderate usage rather than extreme performance. And in the vast majority of applications, this is perfectly reasonable. Even Firefox for example does very well with it.
With that said, new malloc is expected to be somewhat faster than old on lots of workloads (and considerably faster than old would be if we fixed the flaws in old that motivated it), but it's not a performance-oriented allocator. If you really want/need that you should probably link jemalloc or similar (and accept all the tradeoffs that come with that). In Rust programs without "unsafe", it may make sense to do that by default.
Thanks for the clear explanation. Looking at the source code, it looks similar to modern allocators, just without the per-thread heaps. (I think all modern allocators use size-class slab allocators for small objects.) Curiously, I don't think the academic community has much literature on hardened allocators. It's been a while since I've worked in the area, but I wasn't aware of any other than DieHard from 2006 [1]. I did some searched on the ACM Digital Library (I love that it's all free right now so I can easily provide links in forums), and the only other thing I could find was FreeGuard from 2017 [2]. Maybe the issue there is that academics who design memory allocators tend to be on the systems side of CS, and such people tend to use raw performance as a part of the evaluation. Better security for a new thing does not show up in a graph. (Even that FreeGuard paper from 2017 claims security with better performance.)
In the non-academic world, I found the one we're discussing, but also Scudo (https://llvm.org/docs/ScudoHardenedAllocator.html). And that's it. If I still worked in the area, I would try to go after scalable hardened allocators. I wonder if there's still some clever stuff we haven't thought of there.
> However musl has the additional constraint of being compatible with small/very-low-memory environments.
How many threads do these have ?
If they only have one thread, they'll use 72x less memory than if they would have 72 threads.
The thing is that if you are using 72 threads you probably would like your application to be 72x faster than if you are using only one. So synchronizing all allocations and killing scalability doesn't solve these users problems.
Most allocators, including jemalloc, tcmalloc, mimalloc, etc. have a "hardened" mode, that people can opt into if they want.
If I'm using Rust like the user in the blog post, double frees are caught at compile-time, so I'd rather not pay for them at run-time.
Can you provide a demonstration for why this is adequate? It seems like it might still possible to get single quotes in the input to be interpreted by the shell by putting a backslash in the input.
See the man page for a POSIX-compliant shell, like dash[0], and find the section on single-quoted strings. For example:
> Enclosing characters in single quotes preserves the literal meaning of all the characters (except single quotes, making it impossible to put single-quotes in a single-quoted string).
Note also the following examples, with double-quoted strings:
$ echo a b c
a b c
$ echo "a" "b" "c"
a b c
$ echo "a""b""c"
abc
$ echo "a" b "c"
a b c
$ echo "a"b"c"
abc
The same concatenation rules apply to single-quoted strings. So, by putting single quotes at the beginning and end of a string, you only need to worry about single quotes within the string. You can "escape" those with '\'', where the first single quote terminates the preceding single-quoted string, the backslash+single-quote pair is a literal unquoted/escaped single quote in the shell, and the final single quote begins single-quoting again for the rest of the string. The three parts are then dequoted and concatenated together back into your original string by the shell.
The biggest problem with Cygwin is that programs linked with Cygwin inherit global state from from a Cygwin installation on the system they're running on. If you want to produce a Windows program that just runs on any system you install it on using Cygwin, it will work right for most users, but if a power user who has Cygwin installed and has their own custom mounts, options (like different binary/text mode settings), etc. tries to run your program, it might break spectacularly. This makes Cygwin a really poor choice for making binaries you want to distribute as standalone programs.
Aside from that, Cygwin tries to hard to be a complete Unix environment on Windows, whereas midipix just gives you enough to use interfaces that were standardized in POSIX as a reasonable, uniform API for all operating systems to provide. Some functions go beyond that, but you don't have to use them. And even some things that are mandatory in POSIX are optional in midipix; as I understand it, you can choose at build time whether you want the overhead of being able to support tty devices (and the associated semantics like job control, signals from the controlling tty, etc.).
The thing that wont work is if you try to mix and match a dll from one installation with binaries from another, but I think you can agree that that situation is fair. You just need your paths setup correctly.
These are some good observations. Note that on modern POSIX, calling anything but async-signal-safe functions after forking in a multi-threaded process results in undefined behavior. I think it's totally reasonable to consider any programs using the WinAPI (not just POSIX functions provided through midipix) as being formally multi-threaded, and to consider the WinAPI non-AS-safe. I'm not 100% sure what midipix is doing in this regard, but my advice on the project (I'm the primary author/maintainer of musl libc, which midipix is using) has been not to worry about making arbitrary functions work after fork, but only supporting the things that POSIX requires to work. My view in general is that fork should be phased out, but posix_spawn is not sufficiently powerful yet to replace all uses of fork+exec -- it can't do advanced uid changes, setsid, resource limits, etc., and posix_spawn can't be used effectively from an async signal context because the API (attributes and file actions) inherently involves allocation.
No, OOM killer is completely orthogonal to this and is a consequence of not doing correct commit accounting. With strict commit accounting turned on (vm.overcommit_memory=2) fork will correctly fail when there is not sufficient physical backing. But you're correct that fork+exec is a bad model.
The OOM killer exists because of memory overcommitment, which exists because of fork/exec. The justification for overcommitment is that a big-ass process might fork just to do an exec immediately afterwards. If that is the case, then it would be lame to error out the fork because there's not enough room for a second copy of it. But if it doesn't actually exec, then suddenly there's not actually as much memory as it thinks it has as the forked process starts modifying things. This justified overcommitment in the first place and then it snowballed from there.
No, fork is only one path that can lead to overcommit. Allocation of new memory as COW references to a zero page, and COW writable MAP_PRIVATE mappings of files (such as the writable LOAD segments of any executable or library file) also lead to overcommit unless you do proper commit accounting. Any system that does not need to do detailed commit accounting to avoid overcommit is basically wasting the fact that it has virtual memory/MMU.
I never meant to suggest that fork is the only path that leads to overcommitment, but that fork/exec generally insists on overcommitment for reasonable use.
Yes, this is the really big deal that everyone focused on fork and mmap semantics and other details is overlooking. Having midipix as the means of producing Windows versions of cross-platform software like Firefox should make it possible to remove a lot of ugly #ifdeffery and/or whole "portability layers" that are avoiding the standard functions like fopen because of a lack of Unicode support on Windows.
Actually no... the application makes all calls using utf-8, and is expected to provide it in a well-formed manner so that the system call layer could convert it to utf-16. In the reverse route, where utf-16 is read by the system call layer and then converted to utf-8 (getdents(2) and friends), it is expected that file names be in well-formed utf-16. For a file-system volume to have ill-formed utf-16 name entries would make for an interesting case... have not encountered that yet, but will certainly look into that.
Interix is a lot different because it requires installing a system component, which requires administrator privileges. Midipix produces applications that (at least as I understand it) run on basically any NT-based Windows with no special privileges.
I have to lookup what exactly mimalloc is/does every time someone mentions it, because the readme/documentation isn't very descriptive except discussing extensions outside the normal API. I didn't have time to dig through this again today. But I did look at it in some depth on several occasions in the past and it really wasn't suitable for or comparable to what we're doing in musl.