Stream: t-libs/wg-allocators

Topic: Allocators with context


Federico Mena Quintero (May 29 2019 at 23:16, on Zulip):

Hi, everyone. I'm doing an experiment in porting parts of bzip2 (the unmaintained / ubiquitous compression library) to rust. In there, a bz_stream is a C struct that is the main object one interacts with. It lets one define a custom allocator by letting the caller set bz_stream->bzalloc and bz_stream->bzfree fields. Internally the library does a few allocations with those, or falls back to malloc()/free() if the caller set those fields to NULL.
I'm looking in codesearch.debian.net to see just what uses a custom allocator with bzip, and it's only low-level stuff like freetype, ImageMagick and python.
Let's say I end up having a libbzip2.a staticlib built from Rust that preserves the C API. How might I define a Rust allocator that calls into those ->bzalloc and ->bzfree?

Federico Mena Quintero (May 29 2019 at 23:18, on Zulip):

I think I cannot just set the global allocator, because there's no place to hook into the caller-supplied bz_stream

Jake Goulding (May 30 2019 at 19:23, on Zulip):

You can "easily" define your own allocator by implementing the Alloc trait. It does require nightly Rust though.

Jake Goulding (May 30 2019 at 19:24, on Zulip):

The problem comes about with using it, as I don't think any Rust types expose a parameterized allocator.

Jake Goulding (May 30 2019 at 19:25, on Zulip):

RawVec does, but I don't think it's public in any fashion.

Jake Goulding (May 30 2019 at 19:26, on Zulip):

That being said, I bet you don't actually need to make an allocator (although it would be nice)

Tim Diekmann (May 30 2019 at 19:27, on Zulip):

RawVec is feature gated and has an unstable API. I wouldn't use it.

Jake Goulding (May 30 2019 at 19:28, on Zulip):

A thorough answer would require knowing what kind of allocations you fully need, but you can write your own specific code.

Tim Diekmann (May 30 2019 at 19:28, on Zulip):

Also Alloc will probably be changed as well.

Jake Goulding (May 30 2019 at 19:31, on Zulip):

@Tim Diekmann certainly possible! That being said, @Federico Mena Quintero might be in a good position to help explore some of this API from an angle we might not normally get.

Jake Goulding (May 30 2019 at 19:34, on Zulip):

It might also be worth exploring if bzip's NULL means "use malloc" or "use some default".

Jake Goulding (May 30 2019 at 19:43, on Zulip):

You can get reasonably far with appropriate Deref structs though:

use libc; // 0.2.51
use std::{
    ffi::c_void,
    ops::{Deref, DerefMut},
    slice,
};

type AllocFn = unsafe extern "C" fn(usize) -> *mut c_void;

#[derive(Debug, Default)]
struct BzipConfig {
    allocator: Option<AllocFn>,
}

impl BzipConfig {
    fn allocator(&self) -> AllocFn {
        self.allocator.unwrap_or(libc::malloc)
    }

    fn allocate_bytes(&self, size: usize) -> DataChunk {
        let alloc = self.allocator();
        unsafe {
            DataChunk {
                data: alloc(size) as *mut u8,
                cap: size,
                size: 0,
            }
        }
    }
}

// TODO: implement Drop!
struct DataChunk {
    data: *mut u8,
    size: usize,
    cap: usize,
}

impl Deref for DataChunk {
    type Target = [u8];

    fn deref(&self) -> &Self::Target {
        unsafe { slice::from_raw_parts(self.data, self.size) }
    }
}

impl DerefMut for DataChunk {
    fn deref_mut(&mut self) -> &mut Self::Target {
        unsafe { slice::from_raw_parts_mut(self.data, self.size) }
    }
}

fn main() {
    let bz = BzipConfig::default();
    let mut d = bz.allocate_bytes(123);
    println!("{}", d.len());
}
Federico Mena Quintero (May 30 2019 at 21:08, on Zulip):

thanks, I'll see if Alloc does what I need. bzlib only ever uses the custom allocator for "allocate me space for my private structs", and for a couple of arrays. I may be able to ignore the caller's wishes and use whatever allocator Rust wants to use :)

Jake Goulding (May 30 2019 at 23:03, on Zulip):

I'd certainly go the "ignore" route to start with and get the bulk of the functionality in place.

Last update: Nov 15 2019 at 11:15UTC